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(57) Abstract 

A foveated imaging 
system, which can be 
implemented on a general 
purpose computer and greatly 
reduces the transmission 
bandwidth of images has 
been developed. This 
system has demonstrated 
that significant reductions in 
bandwidth can be achieved 
while still maintaining 
access to high detail at any 
point in an image. The 
system is implemented 
with conventional computer, 
display, and camera 
hardware. It utilizes novel 
algorithms for image coding 
and decoding that are 
superior both in degree 
of compression and in 
perceived image quality 
and is more flexible and 
adaptable to different 
bandwidth requirements and 
communications applications 
than previous systems. The 
system utilizes novel methods 

of incorporating human perceptual properties into the coding and decoding algorithms providing superior foveation. One version of the 
system includes a simple, inexpensive, parallel pipeline architecture, which enhances the capability for conventional and foveated data 
compression. Included are novel applications of foveated imaging in the transmission of pre-recorded video (without eye tracking), and in 
the use of alternate pointing devices for foveation. 
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Foveated Image Coding System and Method for Image 
Bandwidth Reduction 



Related Applications : 

This application claims priority under U.S. Title 35 § 119 from 
provisional application No. 60/034,549, filed January 7, 1997, in the 
United States, and provisional application no. 60/035,765, filed January 6, 
1997, in the United States. 

1.0. Rights in the Invention : 

The United States government owns rights in the present invention 
pursuant to grant numbers AF94T004 and F49620-93-1-0307 from the Air 
Force Office of Sponsored Research. 

1.1. Field of the Invention : 

The present invention relates generally to the field of image data 
compression. More specifically, it relates to a foveated imaging system 
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which can be implemented on a general purpose computer and which 
greatly reduces image transmission bandwidth requirements. 

1.2. Description of the Related Art : 

The human visual system provides detailed information only at the 
point of gaze, coding progressively less information farther from this 
point. This provides an efficient means for the visual system to perform 
its task with limited resources. Processing power is thus devoted to the 
area of interest and fewer neurons are required in the eye. Remarkably, 
we rarely notice the severely degraded resolution of our peripheral visual 
field; rather, we perceive the world as a single high resolution image that 
can be explored by moving our eyes to regions of interest. Imaging 
systems utilizing this fact, termed foveation, greatly reduce the amount of 
information required to transmit an image and therefore can be very 
useful for a number of applications including telemedicine, remote control 
of vehicles, teleconferencing, fast visual data base inspection, and 
transmission of pre-recorded video. 

Recently, there has been substantial interest in such foveated 
displays. The U.S. Department of Defense has studied and used so-called 
,f area-of-interest ,! (AOI) displays in flight simulators. These foveation 
schemes typically consist of only 2 or 3 resolution areas (rather than the 
superior continuous resolution degradation) and the central area of high 
resolution is often quite large, usually between 18° and 40° (see, for 
example, Howard, 1989; Warner, et a/., 1993). Other researchers have 
investigated continuous, variable-resolution methods using a log polar 
pixel configuration (Weiman, 1990; Juday and Fisher, 1989; Benderson et 
al. y 1992). The log polar configurations are particularly advantageous 
when rotation and zoom invariance are required, but their 
implementations have necessitated special purpose hardware for real-time 
operation. 
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Juday and Sampsell (U.S. Pat. No. 5,067,019) describe an apparatus 
which will encode a number of space-variant schemes. Fisher later refined 
the apparatus (U.S. Pat. No. 5,208,872). While otherwise effective, these 
systems are limited in that a specific hardware apparatus is necessary to 
perform the foveation operations. Further, these systems have utilized an 
Archimedes spiral to represent the falloff function applied to the image in 
their descriptions of the foveation method. While this may be 
computationally efficient for their specific hardware implementation, it 
does not as accurately represent the actual resolution falloff parameters 
of the human visual system, and does not provide the degree of flexibility 
in controlling the resolution function as the methods proposed here. 
Optimal compression and image quality are obtained by closely 
representing the actual resolution falloff parameters of the human visual 
system. The system of Weimans (U.S. Pat. No. 5,103,306), is more closely 
related to the apparatus described herein. However, implementation of 
the Weimans system requires specific hardware, and the foveation occurs 
in log polar coordinates, rather than Cartesian coordinates. Further, the 
algorithms employed during the compression and reconstruction of the 
foveated image create "cartoon-like" images (page 6, line 52, U.S. Pat. No. 
5,103,306). Wallace, Benderson and Schwartz have also done work in the 
area of space-variant image processing (U.S. Pat. No. 5,175,617). They 
too, used a long polar mapping scheme, but because of the algorithms they 
utilized to perform the compression, the transmission rates were restricted 
to 4 frames per second, well below the rates required for near perceptually 
loss-less encoding. 

2.6. Summary of the Invention : 

The present invention overcomes the limitations in the prior art by 
providing a system that accomplished real-time foveated image 
compression and display, using general purpose computer processors, video 
hardware, and eye-tracking equipment. The foveated imaging system 
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proposed here generates images that are nearly imperceptible from 
uncompressed images to the human observer. Further, the present 
invention makes use of a novel application of modified pyramid coding 
methods for creating foveated images. These coding methods provide 
higher quality images, at high compression rates, with little or no blocking 
or aliasing artifacts. The system can utilize one or more computer 
processors; the number of processors used depends upon the particular 
application and image-processing requirements. 

2.1. TW-Processor FIS With Eye Tracking 

In the inventors' Foveated Imaging System (FIS), eye 
movements are recorded in real time and are used to construct and display 
variable resolution images centered at the current point of gaze (fixation 
point). Typical operation of an exemplar FIS 10 proceeds as follows with 
reference to Fig. 1 for components and Fig. 2 for processes. First, the 
location of an observer 11 fixation point 12 on a display monitor 14 is 
measured with an eye tracking device 16 as depicted in process block 30 
of Fig. 2. Second, the eye position coordinates are transmitted to a remote 
computer 18 as depicted in process block 32. Third, as represented by 
process block 34 of Fig. 2, remote computer 18 captures an image from 
camera 20. Fourth, the camera image is foveated (i.e., encoded and 
compressed so that the resolution of the image decreases from the point 
of fixation) as shown in process block 36. In other words, the degree of 
data compression increases with the distance from the point of fixation. 
Fifth, the encoded image is transmitted by communications channel 21 to 
a local computer 22 as shown in process block 38. Sixth, as depicted by 
process block 40, the received image is decoded and displayed on video 
monitor 14 such that the highest resolution region in the displayed image 
is centered at the fixation point 12. These six steps are repeated 
continuously in a closed loop. The system has been implemented in C ++ 
for execution on standard PC compatible processors, including Intel 
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Pentium® and Pentium® Pro, but other general purpose processors could 
also be used. 

FIS 10 can operate in any one of several different modes. 
The different modes correspond primarily to different methods of foveated 
image encoding and decoding. Mode 1 is the simplest and quickest, and 
hence is most appropriate for large images and/or high video frame rates. 
Mode 2 produces substantially better data compression, for the same 
image quality, but is somewhat slower. Mode 3 produces the greatest data 
compression, but is even slower, and hence is most appropriate for smaller 
image sizes and/or low frame rates. Mode 4 is also slower and produces 
somewhat poorer image quality than Modes 2 and 3; it is included because 
of its wide-spread use. As general-purpose processors become more 
powerful, there will be an increasing number of situations where the high 
compression modes will be appropriate. 

Mode 1: Foveated Pixel Averaging 

In Mode 1, variable size pixels (SuperPixels) are created by 
simple averaging (Kortum and Geisler, 1996). The sizes of the 
SuperPixels increase smoothly away from the point of fixation in a 
manner that matches the decline in resolution of the human visual system 
away from the point of fixation. The color of each SuperPixel is obtained 
by averaging the RGB (or gray level) values of the image pixels that fall 
within the boundary defined by the edges of the SuperPixel. The 
collection of integration boundaries defined by the edges of the SuperPixel 
is called the ResolutionGrid. 

Fig. 3 illustrates one of the ResolutionGrids used in the 
system. ResolutionGrid 50 consists of a series of concentric rings of 
SuperPixels that increase in size away from the point of fixation, Xo, y 0 . 
Because each of the SuperPixels is rectangular, SuperPixels can be 
represented with two pairs of coordinates (which define the summation 
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bounds). This allows fast implementation of operations such as 
summation, clipping, and calculation of ResolutionGrid 50. 

The size of the SuperPixels in the i" 1 ring represented in Fig. 
3 by reference character 52, is calculated according to the formula, 

wherein W, is the size (in screen pixels) of the SuperPixels in ring 52, W B 
is the size of the central foveal SuperPixel 54 (in screen pixels), Xo, y G are 
the coordinates of the fixation point 56 (in degrees of visual angle), x^ 
are the coordinates of the nearest SuperPixel 58 in ring 52 (the i^ ring) 
(in degrees of visual angle), and e 2 is the half-resolution constant (in 
degrees of visual angle). This formula is based on available perceptual 
data, and is also consistent with anatomical measurements in the human 
retina and visual cortex (Wilson et a/., 1990; Geisler and Banks, 1995; 
Wassle et al., 1992). For example, when % is 2.5, the size of SuperPixel 
54 is approximately proportional to the human resolution limit at each 
eccentricity. Thus, if W 0 is less than or equal to the foveal resolution limit 
then the foveated image should be indistinguishable from the original 
image (with proper fixation). If W 0 is greater than the foveal resolution 
limit, then the foveated image should be distinguishable from the original 
image. However, because W 0 is a proportionality constant in equation (1), 
the SuperPixel size will be above the resolution limit by a constant factor 
at all eccentricities. This should distribute any visible foveation artifacts 
uniformly across the display. Increasing % increases the degree of 
foveation and hence the amount of compression. 

Once ResolutionGrid 50 has been computed, a SuperPixel 
averaging subroutine averages the colors of each of the screen pixels that 
fall within each SuperPixel boundary, and assigns the resulting average 
value .to that SuperPixel. SuperPixels at the edge of the display, and at 
the corners of the rings, are either clipped or merged in order to exactly 
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cover the input image. SuperPixels of width 1 are excluded from the 
averaging subroutine. 

Once the SuperPixels have been calculated, conventional 
secondary image compression algorithms are applied prior to transmission, 
decoding, and displaying. An example of a foveated image, without 
secondary compression is shown in Fig. 4. Fig. 4. is a 1024 x 1024 
foveated image (Mode 1) with fixation at point 59 on the license plate of 
the Honda CRX. Because the pixel size is below the resolution limit of the 
visual system (for this size image), the resulting degradations due to 
foveation are imperceptible with appropriate fixation. The blockiness of 
the tree provides verification that this is a foveated image. 

Mode 2: Foveated Laplacian Pyramid 

Modes 2 and 3 make use of pyramid coding/decoding 
methods. Pyramid methods are widely used in non-foveated image 
compression (Burt & Adelson, 1983; Adelson, Simoncelli & Hingorani, 
1987; Shapiro, 1993; Strang & Nguyen, 1996; Said & Pearlman, 1996). 
One new contribution of the Inventors' FIS is the modification of pyramid 
algorithms to include foveated image compression. Another contribution 
is the discovery that foveation not only increases the amount of image 
compression obtainable with pyramid methods, but substantially increases 
the speed of pyramid calculations. This makes foveated pyramid methods 
more feasible for real time video compression than non-foveated pyramid 
methods. 

In Mode 2, a modified Laplacian pyramid algorithm (Burt & 
Adelson, 1983) is utilized. A simple example of the Foveated Laplacian 
Pyramid (FLP) algorithm is illustrated in Fig. 5. The pixels of an input 
image 60 are represented by the large grid in the upper left corner of Fig. 
5. The coding steps are as follows for each level of the pyramid as shown 
in Fig. 5. 
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REDUCE. First, starting image 60 is filtered by convolution 
with a linear weighing function (kernal), and then down-sampled by a 
factor of 2 in each dimension, to obtain a smaller, lower-resolution image 
62. In Fig. 5, the filter kernal is a 2 x 2 block, each element having a 
weight of 1/4; thus, this REDUCE operation is equivalent to computing 2 
x 2 SuperPixels over the entire image (as in Mode 1). For example, b n is 
the average of a n , a 12 , and a^. Other kernals may also be used as later 
described in section 4.2. 

EXPAND. Next, down-sampled (reduced) image 62 is 
interpolated (up-sampled) to obtain a larger, lower-resolution image 64. 
A great computational benefit of foveation is that it is usually only 
necessary to perform this EXPAND operation on a subset of the elements 
in the reduced image; i.e., on those elements within some distance of the 
fixation point, Xq, y 0 . The elements over which the EXPAND operation is 
applied is determined from formulae derived from human perceptual data 
(see later) and from the current fixation position. Importantly, the 
percentage of elements over which the EXPAND operation needs to be 
applied is smallest for the largest reduced images (see later section 4.2). 

DIFFERENCE. Next, expanded image 64 is subtracted from 
the starting image 60 to obtain a difference image 66. Again, there are 
computational savings over non-foveated pyramid methods since the 
DIFFERENCE operation is only applied to the image region that was 
expanded. 

THRESHOLD. As indicated below, difference image 66 is 
thresholded, based upon formulae derived from human perceptual data 
(see 4.2). In the FIS, the threshold is a function of both the spatial 
frequency "passband for thecurrent pyramid level, and the distance to the ! 
fixation point. It is at this step that much of the image compression due 
to foveation is obtained. Again, there are also computational savings over 
non-foveated pyramid methods since the THRESHOLD operation is only 
applied to the image region that was expanded. 
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QUANTIZE. Next, the thresholded image is quantized to 
obtain additional data compression. Quantization is a form of lossy 
compression where the number of colors (or gray levels) used to represent 
image elements is reduced. In the FIS, the level of quantization is a 
function of both the spatial frequency passband for the current pyramid 
level, and the distance to the fixation point. In this step, further image 
compression due to foveation is obtained. Again, there are also 
computational savings over non-foveated pyramid methods since the 
QUANTIZE operation is only applied to the image region that was 
expanded. 

LOSSLESS COMPRESSION. The final step in the coding 
process is standard lossless compression (e.g., Huffman or arithmetic 
coding, image differencing across frames). Again, there are also 
computational savings over non-foveated pyramid methods since 
LOSSLESS COMPRESSION is only applied to the image region that was 
expanded. 

As shown in Fig. 5, all of the above operations are repeated 
for as many levels of the pyramid as desired. In the current FIS, all of the 
REDUCE operations are performed first. Computation of the thresholded, 
quantized and lossless-compressed difference images then begins with a 
smallest (lowest resolution) reduced image and proceeds toward the 
largest (highest resolution) reduced image. 

TRANSMISSION. All of the coded difference images, and 
the final reduced image are transmitted to the local computer 22 (see Figs. 
1 & 2), beginning with the final reduced image, and working backwards 
to the level-1 difference image. Each coded difference image is 
transmitted as soon as it is computed. 

DECODING. Following transmission, the image data is 
decoded and displayed. The decoding proceedings in the reverse order of 
the coding. (1) Reverse the lossless compression. (2) Expand the quantize 
and thresholded data levels to the appropriate range. (3) Decode the 
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pyramid to create the final transmitted image. During coding, it is only 
necessary to EXPAND those elements within some distance of the fixation 
point, Xo,y 0 . However, for decoding it is necessaiy to apply EXPAND all 
of the elements at each level. 

Mode 3: Foveated Wavelet Pyramid 

Mode 3 utilizes a modified subband wavelet pyramid in place 
of the modified Laplacian pyramid. The processing steps in the Foveated 
Wavelet Pyramid are very similar to those of the Foveated Laplacian 
Pyramid; thus only the differences are described. 

ANALYZE. In the wavelet pyramid the REDUCE, EXPAND 
and DIFFERENCE operations at each level of the coding pyramid are 
replaced by an ANALYZE operation which involves convolution with two 
or more linear kernals followed by downsampling. The most common 2D 
wavelet pyramids are based upon convolution with a low pass kernal, l(x)> 
and a high pass kernal h(x) y followed by down-sampling by a factor of 2 
in each dimension (as in the Laplacian pyramid). The result, at each level 
of the pyramid, is four subband images: LL (low pass in both directions), 
HL (high pass in the x direction and low pass in the y direction), LH flow 
pass in the x direction and high pass in the y direction) and HH (high 
pass in both directions). 

LL = l(x)*(Ky)*i(xjr)) sampl(x,y) (2) 
HL = h(x)*(l(y)*i(x,y)) sampi(x,y) (3) 
LH = l(x)*(h(y)*i(x,y)) sampt(x,y) (4) 
HH = h(x)*(h(y)*i(x,y)) sampi(x,y) (5) 
wherein i(x,y) represents the input image (or subimage), sampl (x,y) 
represents the sampling function (which sometimes must be different for 
each of the four equations), and * represents the operation of filtering 
(convolution). These four subimages together contain exactly the total 
number of elements in the input image. 
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The three high-pass subimages (HL, LH, HH) play the role 
of the difference image in the Laplacian pyramid. To them are applied the 
THRESHOLD, QUANTIZE and LOSSLESS COMPRESSION operations, 
prior to transmission. The strictly low-pass subimage (LL) plays the role 
of the reduced image in the Laplacian pyramid, it is used as the input 
image to the next level of the pyramid (e.g., it is processed by equations 
2-5). The subimages for the first two levels of a wavelet pyramid are 
illustrated in Fig. 6. 

The elements over which the high-pass convolutions are 
applied are determined from formulae derived human perceptual data (see 
later) and from the current fixation position. Because of the resolution 
properties of the human visual system, the percentage of elements over 
which the high-pass convolutions need to be applied is smallest for the 
largest subimages. This computational savings greatly increases the speed 
of the wavelet pyramid calculations, since only for the LL image (equation 
2) must all the elements be processed. 

SYNTHESIZE. Following transmission and initial decoding 
(undoing the LOSSLESS COMPRESSION, THRESHOLD and 
QUANTIZE operations) the subband images are synthesized into the final 
image which is then displayed. The SYNTHESIZE operation is similar to 
the final EXPAND and ADD operations of the Foveated Laplacian 
Pyramid. Specifically, the highest-level subimages are upsampled, filtered 
by a pair of low-pass and high-pass synthesis kernals and then added 
together. The resulting image and the subimages at the next level are 
then upsampled, filtered by a pair of low-pass and high-pass synthesis 
kernals and then added together. This process repeats until the full image 
is synthesized. 

There are many possible choices for the low and high-pass 
kernals. Examples, are the 9-tap QMF filters of Adelson, Simoncelli & 
Hingorani (1987) or the 9/7-tap filters of Antonini, Barlaud & Daubechies 
(1992). 
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Mode 4: Foveated Block DOT 

Mode 4 utilizes standard block DCT (Discrete Cosine 
Transform) coding/decoding rather than pixel averaging or pyramid 
coding. In block DCT the image is divided into non-overlapping blocks; 
a discrete cosine transform is then applied to each block (Stang and 
Nguyen, 1996). For example, in the JPEG standard the block sizes are 8x8 
pixels. In Foveated Block DCT, the DCT coefficients are thresholded, 
quantized, lossless coded and transmitted in a fashion similar to that for 
the Foveated Pyramids. The DCT coefficients that need to be computed 
are determined from formulae derived from human perceptual data (see 
later) and from the current fixation position. Because of the resolution 
properties of the human visual system, the number of coefficients that 
needs to be computed decreases drastically away from the point of 
fixation. This computational savings greatly increases the speed of block 
DCT coding and decoding. 

2.2. Eye Position Tracking 

After initialization of parameters (and eye tracker calibration, 
if necessary), the system enters a loop in which the position of the eye is 
measured (see block 30 of Fig. 2). The measurement of eye position is 
done with a commercial eye tracking device 16. The current system uses 
a Forward Technologies Dual Purkinje Eyetracker, but almost any 
commercially available eye tracker would do. Eye position is measured 
and transmitted to the remote processor at approximately the frame rate 
of the display monitor. In Mode 1, the eye position is used to compute a 
new ResolutionGrid 50 centered on the point of fixation. In Modes 2 and 
3, the eye position is vised to set the region over which image elements are 
processed, and to set the threshold and quantization functions (see later). 

Because of noise in the eye tracker and because of small 
fluctuations in eye position, it is usually desirable not to change the 
fixation point (x^o) used in the coding algorithm unless the change in 
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measured eye position exceeds a threshold (e.g., 0.5 deg of visual angle). 
The rule used in the current system is: 

if ^(xo-O 2 + (yo-yj 2 <6 then leave (Wo) unchanged, 

otherwise set (x^y,) equal to (x^J (6) 
where (x^ J is the current measured eye position and 6 is a threshold. 

2.3 Four-Processor FIS with Eye Tracking 

The process of foveation greatly reduces the amount of image 
data that needs further processing, and thus makes possible simple, 
inexpensive parallel pipeline architectures such as that in Fig. 7. The four 
processor system 70 illustrated in Fig. 7 consists of two remote processors 
72 and 74 (e.g., two Pentium® computers) and two local processors 76 and 
78. One of the remote processors 72 performs the initial foveation coding, 
while the other processor 74 performs the additional lossy and lossless 
compression, plus the transmission over the main communication line 21 
to the local processors 76 and 78. One of the local processors 76 receives 
the remote data and performs the initial decoding into foveated image 
data, while the other processor 78 expands and displays the foveated 
image data* Because of the substantial data compression due to the 
foveation stage, simple, inexpensive, low bandwidth communication 
channels (see later) 80A and 80B are all that is needed between the two 
remote computers and between the two local computers. This system is 
capable of real time operation with larger image sizes and/or higher frame 
rates than the two-processor system 10. The only specialized piece of 
hardware required is a simple parallel communication card (or 
communications card of equivalent speed) in each computer. 

This system 70 is particularly well adapted for Modes 2 and 
3. As each difference image, or subband image, is computed in first the 
remote computer, it is transmitted to the other remote computer for 
thresholding, quantization, lossless coding and transmission. Similarly, 
as each difference or subband image is decoded by the first local computer, 
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it is transmitted to the other local computer for expansion/synthesis and 
display. 

2.4 FIS Without Eye Tracking 

In general, foveated images appear low in quality unless 
fixation is at the proper position. Thus, for most imaging applications, eye 
tracking is an essential component of a useful foveated imaging system. 
There are, however, two exceptions. 

One exception is when a mouse or some other pointing device 
can be used in place of the eyes. This is a simple modification of the 
system where the screen coordinates from the alternate pointing device 
are transmitted to the remote computer rather than the screen 
coordinates from the eye tracker. In all other respects, the system would 
be the same as described earlier. Alternate pointing devices could be 
useful in situations where eye tracking would be impossible or too 
expensive, yet not absolutely essential. Some examples would be 
surveillance systems or systems for inspecting/searching large data bases 
of images (e.g., satellite imagery). In these applications, pointing-device 
control of foveation might be adequate. In some cases, the remote control 
of vehicles, foot controls, touch pads, or pointing sticks would be adequate 
for controlling the foveation point. 

The other exception is the presentation of pre-recorded video. 
In static images, observers typically fixate on a large number of points 
within the image, making observer-controlled foveation essential. 
However, in dynamic video situations, observers tend to fixate at certain 
critical points within the images. Thus, it might be possible to transmit 
foveated images based upon the most likely point of fixation, without 
observer-controlled foveation. Unfortunately, there are no adequate 
algorithms or models for predicting where observers will fixate in dynamic 
(or static) video sequences. However, for prerecorded video (e.g., movies), 
it would be possible to simply measure, in prior testing, where observers 
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fixate the unfoveated video sequence. In other words, an eye tracker 
would be used in pre-testing to measure the point (or points) of fixation 
in every frame of the video, for each test observer. The fixation patterns 
of the test observers would then be used to create a foveated version of 
the pre-recorded video, using algorithms similar to those described in this 
document. The foveated version of the pre-recorded video would be 
transmitted to the user. The most obvious application of this type of 
foveated imaging system would be in transmission of pre-recorded video 
for high-definition digital television (HDTV). In this application, where 
very high perceptual quality is required, the amount of foveation would 
presumably be kept relatively small. Nonetheless, even a factor of two in 
additional compression (from foveation) would be of great significance. 

i 

2.5 Key Design Features of the Foveated Imaging System 

The present invention contains many key design features. 
First, it can be implemented with conventional computer, display, and 
camera hardware, insuring low cost and easy improvements to the system. 
Most, if not all, previous systems that operate at useful video frame rates 
involve special purpose hardware that is costly and quickly becomes 
obsolete. 

Second, the system utilizes novel algorithms (Foveated 
Laplacian Pyramids, Foveated Wavelet Pyramids and Foveated Block 
DCT) for image coding and decoding. These algorithms produce very high 
compression rates with high perceptual image quality. They are superior 
both in degree of compression and in perceived image quality to previous 
foveated imaging algorithms. Furthermore, Foveated-Laplacian-Pyramid, 
Foveated-Wavelet-Pyramid and Foveated-Block-DCT algorithms produce 
greater compression, and have faster execution times, than the non- 
foveated versions of these algorithms. The present system (unlike 
previous foveated imaging systems) makes practical real-time use of these 
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highly-desirable coding algorithms on conventional, inexpensive, computer 
hardware (e.g., Pentium® class PCs). 

Third, the system is designed to be highly flexible and 
adaptable to different situations. The overall display resolution, as well 
as the coding algorithm (the mode), is adjustable to accommodate different 
communication channel bandwidths and different image-size/frame-rate 
requirements. In addition, variable resolution, quantization and 
thresholding parameters are adjustable by the user to optimize subjective 
appearance or task performance. 

Fourth, the system utilizes novel methods of incorporating 
human perceptual properties into the coding and decoding algorithms. 
Specifically, the system combines known variations in human contrast 
sensitivity, as function of spatial frequency and retinal eccentricity, in 
order to closely match the image compression to the image-coding 
properties of the human visual system. A close match insures that 
maximum compression is achieved with optimal perceptual quality. No 
previous system uses both smooth foveation based upon the decline in 
spatial resolution with eccentricity, and thresholding/quantization based 
upon the variation in contrast sensitivity with spatial frequency. The 
system's default variable-resolution algorithms, based upon quantitative 
(psychophysical) measurements of the variations in human contrast 
sensitivity with eccentricity and spatial frequency, provide superior 
foveation. 

Fifth, the simple parallel pipeline architecture described in 
2.3 and 4.5 enhances the capability for conventional data compression in 
conjunction with foveated compression. The represents an inexpensive 
method of increasing the image sizes and/or frame rates, and the level of 
image compression that can be obtained. 

Sixth, the use of a foveated imaging system with alternate 
pointing devices, and in the transmission of pre-recorded video (without 
eye tracking), are novel applications. 
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The list of potential applications for the invention include: 

• Teleconferencing (requires two foveated imaging systems) 

• Remote control of vehicles 

• Telemedicine 

• Surveillance 

• Fast visual data base inspection 

• Transmission of pre-recorded video 

3.0 Brief Description of the Drawings : 

Fig. 1 depicts a preferred embodiment of a two-processor Foveated 
Imaging System. 

Fig. 2 illustrates a general flow diagram of the operation of the two- 
processor Foveated Imaging System depicted in Fig. 1. 

Fig. 3 illustrates a Foveated imaging system SuperPixel pattern 
arrangement called a ResolutionGrid. 

Fig. 4 is a 1024 x 1024 foveated image (Mode 1). 

Fig. 5 is an examplar of how a Foveated Laplacian Pyramid may be 
computed. 

Fig. 6 is a depiction of two levels of a Foveated Wavelet Pyramid. 

Fig. 7 depicts a preferred embodiment of a four-processor Foveated 
Imaging System. 

Fig. 8 is a block diagram of PCI bus local communications card. 

Fig. 9 is a general flow diagram for the C version of the Foveated 
Imaging System operating in Mode 1 (Foveated Pixel Averaging). 

Fig. 10 is a general flow diagram for the C* + version of the 
Foveated Imaging System operating in Mode 1 (Foveated Pixel Averaging) . 

Fig. 11A graphically represents the process of creating the 
SuperPixel list in a preferred embodiment of the invention. 

Fig. 11B graphically shows the process of averaging pixels from the 
input image. 
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Fig. 11C graphically illustrates the process of display in a preferred 
embodiment. 

Fig. 12 depicts Foveated Laplacian Pyramid encoding. 

Fig. 13 is a diagram of Foveated Laplacian Pyramid decoding. 

Fig. 14 depicts a 2 x 2 REDUCE operation. 

Fig. 15 depicts a 3 x 3 REDUCE operation. 

Fig. 16 depicts a 2 x 2 EXPAND operation. 

Fig. 17 depicts a 3 x 3 EXPAND operation. 

Fig. 18 further illustrates a 2 x 2 version of the EXPAND operation. 

Fig. 19 further illustrates a 3 x 3 version of the EXPAND operation. 

4.0 Description of Preferred Embodiments 

An important facet of the current invention is its ability to 
be run with conventional hardware, including computer, display, camera 
and eye-tracking equipment. In one embodiment of the invention, the 
computers may be any PC compatible Pentium® or Pentium® Pro with a 
PCI bus. Acceptable cameras include any NTSC (RS-170) camera (e.g., 
Sony XC-75) and many digital cameras (e.g. Dalsa CA-D2-512 digital 
camera, Dalsa, Waterloo, ONT, Canada). Conventional frame grabbers are 
also sufficient (e.g., MuTech MU-1000/MT-1300 (RS-170) or MU-1000/MV- 
1100 (digital), MuTech, Billerica, MA) as camera 20. Almost any 
commercially available eye-tracker 16 can be used in the system (e.g., 
Forward Technologies, San Marcos, TX; ISCAN Inc., Cambridge MA; 
Applied Science Laboratories, Bedford, MA; Express Eye, Renquishausen, 
Germany). To read the eye tracking signals, any A/D card (12 bits or 
better) and digital I/O card (8 bits or better), will suffice (e.g., Analogic 
HSDAS-16, Analogic, Wavefield, MA). 

The only more specialized hardware would be a simple local 
communication interface card (able to support a transfer rate of 3 
mbytes/sec) needed with the Four-Processor FIS system 70 (see 2.2 and 
Fig. 7). This communication card is not required for the Two-Processor 
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FIS system 10. A block diagram of one such card is shown in Fig. 16. 
The description of the blocks of Fig. 8 is as follows. 

PCI Bus interface 100 — Implements capabilities required to 
attach a device to PCI bus 102 - including configuration registers, and a 
two quad-byte address space. The lower address represents the DATA in 
and out registers, the higher the CONTROL in and out registers. PCI 
Bus interface 100 also implements a 32 bit bi-directional data bus with a 
1 bit address ,f bus" and read/write strobes for communicating with Data 
latches 104 and 105 and 3-state buffers 106. 

Address and Read/Write decode logic 110 — Turns the PCI 
Bus interface 100 address bus and read write strobes into separate, 
address specific read and write strobes (read control, write control). 

Data and Control output latches 104 and 105 hold Data 
written by PCI Bus interface 100, and make it available to the line drivers 
112A and 112B for transmission to the other computer. 

Data-in buffers 114 and Control-in 3-state buffers 116 pass 
data from bus receivers 117A and 117B (and from the other computer) to 
PCI Bus interface 100 in response to read strobe signals from the PCI Bus 
interface 100 (through the Address and Read/Write decode logic). 

DAV logic block 118, together with the ACK logic block of 
the other computer implement handshaking between computers. DAV 
Logic 118 generates a DATA AVAILABLE (DAV) signal to notify the 
receiving computer that data has been placed on the data out lines and 
should be read. The DATA AVAILABLE signal remains asserted until the 
receiving computer asserts the DATA ACKNOWLEDGE (ACK) signal line, 
signifying that the transmitted data has been read. The DAV signal is 
also connected to Control-In 3-state buffers 116 to allow the program 
controlling the transmission of data to determine when a previously 
transmitted data has been safely received. 

ACK Logic 120 implements the receiving end of the 
DAV/ACK handshaking. Receipt of a DAV signal notifies the controlling 
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program, through the Control-In section, that data is available and must 
be read. When the controlling program reads the Data-in register, ACK 
Logic 120 asserts the ACK signal. When the other computer sees the ACK 
signal, it de-asserts it's DAV signal, which causes ACK logic to de-assert 
its ACK signal. The system is then ready for the next transmission. 

Line drivers and receivers will depend on transmission 
distance requirements. For short distances and lower data rates, none 
will be needed. For longer distances and higher data rates, more 
sophisticated drivers and receivers can be used. 

4.1 Foveated Pixel Averaging 

One version of the system, with Foveated Pixel Averaging, 
has been implemented in C (Portland Group PGCC) for execution on an 
ALACRON 1860 (Alacron Inc., Nashua, NH) processor, but better 
performance can be obtained on a Pentium® Pro processor. Fig. 9 
illustrates the general function of the system. Upon system initialization 
depicted by block 200, the user is queried for a number of required 
parameters (pixel dimensions, image size, image distance, half resolution 
constant, eye movement update threshold, etc.). Using these parameters, 
a space variant arrangement of SuperPixels (referred to as the Resolution 
Grid) is then calculated and stored for display as shown by process block 
202 (a SuperPixel is a collection of screen pixels that have been assigned 
the same gray level value). Next, an eye position calibration is conducted 
shown by process block 204. The system then enters a closed loop mode, 
in which the current eye position is determined and compared with the 
last known eye position as depicted in process block 206. If the eye has 
not moved more than some amount specified during initialization (decision 
block 208), then pixel averaging (based on the Resolution Grid) is executed 
as shown in block 210. However, if the eye has moved more than this 
threshold amount then the coordinates of the ResolutionGrid are offset by 
the magnitude of the eye movement before pixel averaging as depicted by 
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process block 209. Finally, the resulting SuperPixels are sent to the 
display device as shown in block 212, and eye position is checked again. 

Only the closed loop portion of the program is required to 
run in real time. Initialization, calibration and calculation of the space- 
variant resolution grid take place prior to image display. However, 
because of the simplicity of the resolution grid structure (which is 
described in greater detail below), it can also be re-calculated in real time, 
if desired. 

The system has also been implemented in C++ (Watcom 
C/C++) for execution on an Intel Pentium® processor. This 
implementation works similarly to the C version: parameters are entered 
by the user and the system is initialized, and the system then enters a 
closed loop in which eye movement data is collected. Within the closed 
loop, the image is foveated and displayed based upon the movement of the 
eye. 

One difference between the C implementation and the C+ + 
implementation is that whenever the eye moves beyond the eye movement 
threshold, the C+ + implementation recomputes the entire ResolutionGrid 
based upon the new eye position. In the C implementation, the 
ResolutionGrid must be 4 times greater than the image in order to 
facilitate offsetting the grid relative to the fixation point. Each SuperPixel 
in the ResolutionGrid must then be examined to determine whether or not 
it lies within the image boundary. The Superpixels that do not lie within 
the image boundary are then clipped. In the C++ implementation, since 
the Resolution Grid is recalculated with each eye movement (that exceeds 
the eye movement threshold), the grid needs only be as large as the image, 
and, therefore, no clipping is required. In these particular 
implementations, recomputing the ResolutionGrid and not clipping 
SuperPixels has shown a slight performance benefit over computing a 
single ResolutionGrid and offsetting it relative to the fixation point. 
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Fig. 10 illustrates a flowchart description of Foveated Pixel 
Averaging and the processes of creating the Superpixel list, averaging 
pixels from the input image, and displaying them on the display are 
summarized in Figs. 11A, 11B, and 11C, respectively. 

Block 214 in the flow chart of Fig. 10, shows the 
initialization phase. During the initialization phase parameters are 
entered and set as described above. 

After initialization, the current measured eye position (x^J, 
is obtained as shown by blocks 215 and 216. The eye position's distance 
from the systems's fixation point, (x^o), is then compared to a threshold, 

6 in decision block 218. If ^-xj 2 + (yo-yj 2 <& then (x^o) remains 
unchanged, and the "No" branch of the flowchart is taken. Otherwise 
(xoJTq) is set equal to (x^J, and the ! Yes" branch of the flowchart is 
followed in order to create a new list of SuperPixels as depicted by block 
220. 

When a new list of SuperPixels is created, the system's 
fixation point and the parameters given during initialization are used to 
determine each SuperPixel's properties. Each SuperPixel in the list 
specifies a size and starting coordinate relative to the viewed image. The 
final list collectively describes a non-overlapping grid of pixel regions that 
cover the entire image. Creation of the SuperPixel list is graphically 
shown in Fig. 11A. 

The next step in the process is to average pixels in the input 
image in order to get an average color or grey scale value for each 
SuperPixel as shown in block 222. Averaging of grey scale values is done 
by simply summing pixels over the superpixeFs pixel region and dividing 
by the number of pixels in the superpixel. Averaging of unmapped, or 
true-color values is done the same as for grey scales, except the individual 
red, green, and blue intensities are each averaged. Averaging of color 
mapped images (images whose pixel values correspond to indices in a map 
of color values) is achieved by first converting each superpixel's RGB 
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(tristimulus) components to YUV (color sum/difference) components. 
These YUV components are then averaged, and the resulting average YUV 
components are used to find the closest matching color value in the 
image's color map. The averaging process is graphically shown in Fig. 
11B. 

The final step is to write the Superpixels to the display in 
process block 224. The simplest way to do this is to write each pixel as 
a block whose pixels are all the same value as the Superpixel's averaged 
color or grey scale value. A better way is to interpolate pixel values in 
order to create a smooth image that does not contain block edges. In 
order to do this, each Superpixel's grey scale or color value is written to 
the display buffer in the center of that Superpixel's pixel region. Then, 
bilinear interpolation is used to smooth the image. Specifically, pixels are 
linearly interpolated in the vertical direction to create smooth color 
changes between all vertically adjacent superpixel centers. Pixels are then 
linearly interpolated in the horizontal direction to create smooth color 
changes between the vertically interpolated pixels. Constructing an image 
is graphically shown in Fig. 11C. 

4.2 Foveated Laplacian Pyramid 

The Foveated Laplacian Pyramid system has been 
implemented in C ++ (Watcom C/C ++ ) on an Intel Pentium® processor. 

Foveated Laplacian Pyramids work similarly to Laplacian 
Pyramids, however, during the expand operation of the Foveated 
Laplacian Pyramid, only the region within the human contrast threshold 
needs to be Expanded and Differenced during the encoding process. This 
means that during the decoding process, only in this same region will 
differenced images need to be added back to obtain the original image. 
This is illustrated in Fig. 12 and Fig. 13. 
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REDUCE. In the 2 x 2 version, the REDUCE operation 
works as follows. Each 2x2 block in Level I is averaged to obtain a pixel 
in level Z+I. For example, as shown in Fig. 5 and Fig. 14, 

In the 3 x 3 version shown in Fig. 15, the REDUCE 
operation works as follows. Each 3x3 block in Level / is averaged using 
the following weighing function to obtain a pixel in Level Z+l: 
[1/16 1/8 1/16] 

W= [1/8 1/4 1/8] (8) 
[1/16 1/8 1/16] 



For example, 



fll1+2fll2+gl3 + *21 +2*22**23 + ^31 +2*32 "^33 ,Q) 

"~ 16 8 16 

fl 13 +2fl 14 +fl 16 . *23 +2g 24 +fl 25 , g 33+ 2a 34+ fl 3S i 1 A\ 

16 + 8 16 UW 



EXPAND. The EXPAND operation for the 2 x 2 version 
works as follows and as shown in Fig. 16. First note that the solid dots 
at the grid intersection, at the top of Fig. 5, show the appropriate spatial 
location of the down-sampled pixels representing the 2x2 block averages. 
Thus, for example, b u in the Level 2 grid is the local average color (e.g., 
gray level) at the position indicated by the solid dot in the Level 1 grid. 
This is illustrated more fully in Fig. 18, where the Level 2 pixels have 
been overlaid on .the Level 1 grid. 

The purpose of the EXPAND operation is to 
interpolate/expand the reduced image to its size prior to applying the 
REDUCE operation. For each pixel location in the expanded image, the 
nearest three pixels in the reduced image are found. For example, the 
three pixels in the reduced image that are nearest to a^ are b n , b^ and b^. 
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The three nearest pixels define a plane in color (or gray scale) space. The 
interpolated value is obtained by evaluating the color plane at the location 
of the pixel in the expanded image. The algebra of this calculation is to 
first find the average of the two pixels defining each of the perpendicular 
sides of the triangle and then to find the average of those two averages. 
For example, 

*11 + *12.*11 + *21 



2 5 2 - ^ * (ID 



Similarly, 



+ 



2 4 ^2 4 ^2 4 



(12) 



The EXPAND operation for the 3x3 version is simpler. The 
sold dots in Fig. 19 show the appropriate spatial locations of the down- 
sampled pixels representing the 3x3 averages. The interpolated values 
are obtained by bi-linear interpolation. The interpolated pixels between 
the level Z+l pixels are obtained by simple averaging. For example, 

i. . and Jj* - (13) 



The remaining interpolated pixels are obtained by averaging 
the initially interpolated pixels. For example, 



(14) 



(See also. Fig. 18). 

DIFFERENCE. The DIFFERENCE operation is to simply 
subtract the expanded image from the starting image. For example, for 
each location ij in the Level 1 (Fig. 5), 



25 



WO 98/33315 



PCT/US97/23737 



fly a ^ - a v (15) 

THRESHOLD. The color (or gray level) of each pixel, in 
each difference image, is thresholded in a fashion that varies with level in 
the pyramid and with distance from the point of fixation (eccentricity). 
Different threshold functions can be selected based upon the application. 
Here is described a particular threshold function for gray scale images. 
To maximize perceptual image quality the inventors have implemented a 
threshold function which is consistent with measurements of contrast 
sensitivity in the human visual system as function of spatial frequency and 
retinal eccentricity (Wilson et al. 1990; Geisler and Banks, 1995). 

Let Xo,y 0 be the point of fixation in the image expressed in 
degrees of visual angle from the center of the image. If a pixel is located 
at position x, y, then the eccentricity, c, is given by: 

e = ^(*-*b) 2 ♦ (y-yo) 2 (16) 
The threshold function is given by the following equation: 

G T (e,l) = . C T (e,l) (17) 
where is the maximum possible absolute value in the difference image 
(e.g., 128 for 8-bit gray level), and Cr(e,l) is the human contrast threshold 
for the eccentricity, e, and for the spatial frequency passband associated 
with Level Z. By definition the maximum contrast threshold is 1.0. 

The purpose of the threshold is to eliminate all difference 
pixels which are below human contrast threshold. If G is the value of the 
pixel at location in the difference image for level I then the post- 
threshold value is given by, 



I 0 otherwise 



(18) 



Notice that when contrast threshold for a pixel reaches a value of 1.0 then 
difference-image value cannot exceed the threshold, and hence the pixel 
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need not be transmitted. Additional compression is obtained because 
equation (18) reduces the range of difference values. 

In the current version of the algorithm we use the following 
formula for human contrast threshold although other similar formulas 
could be used: 

+ mln |c 0 exp (a j*j .1 j (19) 

where C 0 is the minimum contrast threshold (across all spatial frequencies 
and eccentricities), u x is the dominant spatial frequency of the passband 
of Level I, is the eccentricity at which human spatial resolution falls to 
one-half the resolution at zero eccentricity and a is a constant. The 
inventors have found that this formula provides a reasonable 
approximation to the falloffin the contrast sensitivity of the human visual 
system as a function of spatial frequency n l and eccentricity e. 

Setting C T (e,l) to 1.0 and solving for e yields a formula which 
gives the eccentricity, e^, beyond which it is unnecessary to transmit 
pixel information and unnecessary to perform the EXPAND operation: 

^ . Ji^ - e2 (20) 

For low levels (e.g., Level 1) the dominant spatial frequency, 
u lf is high and hence e^ is small. Thus, the greatest compression and 
computational savings are obtained for the largest size difference image. 

QUANTIZE. Further compression is obtained by 
quantization. Many image compression algorithms make use of the fact 
that humans are able to resolve fewer levels of gray in higher spatial 
frequency bands than in low frequency bands. We also allow the 
possibility of having the level quantization vary with retinal eccentricity. 
If G' is the thresholded value of the difference-image pixel at location xy 9 
then the value after quantization is given by, 

G" = NINT(Q(e,l).G>) (21) 
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where NINT is the "nearest integer" function, and Q(e,l) is the 
quantization function: 

0 s<?(«,1)*1.0 (22) 
LOSSLESS COMPRESSION. The final step before 
transmission is lossless compression. (1) If the point of foveation has not 
changed then, for each level of the pyramid, only the difference between 
the current compressed image and the previous image is transmitted. If 
the point of foveation has changed then frame differencing is not used. 
(2) Huffman or arithmetic coding is applied to the final images for each 
level of the pyramid. 

DECODING. Following transmission, the image data is 
decoded and displayed. The decoding proceeds in the reverse order of the 
coding. (1) Reverse Huffman or arithmetic coding. (2) Add the difference 
image to the previous image (if point of fixation was the same). (3) Divide 
G" by Q(e,l) to expand the gray levels to G' values. (4) Add G T (e,l) to G' 
if G' > 0, subtract G T (e,l) from G' if G'<, 0 to expand gray levels to G 
values. (5) Decode the pyramid to create the final transmitted image. For 
example, decoding of the coded video image data of Fig. 5 would proceed 
as follows: EXPAND the Level 4 image and ADD the result to the Level 
3 difference image to obtain the Level 3 image, EXPAND the Level 3 
image and ADD the result to the Level 2 difference image to obtain the 
Level 2 image, EXPAND the Level 2 image and ADD the result to the 
Level 1 difference image to obtain the Level 1 image. 

Note that for decoding it is necessary to apply EXPAND to 
the entire image at each level (se Fig. 13). Whereas, during coding it is 
only necessary to apply EXPAND within the region defined by e^see Fig. 
12). 

4.3 Foveated Wavelet Pyramid 

The Foveated Wavelet Pyramid may be implemented in C ++ 
(Watcom C/C* + ) on an Intel Pentium® processor. The computations can 
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be of the form typical in the literature, with the addition of the 
modifications for foveated imaging described above. Specifically, the 
THRESHOLD and QUANTIZE operations will be applied to each of the 
subimages. The only difference is that I in equations (17)-(22) will now 
represent a subimage of the wavelet pyramid, and u, will represent the 
dominant spatial frequency of the passband associated with that subimage. 
Equation (20) will be utilized to determine the range of elements for 
which the high-pass filtering (h) needs to be computed. 

4.4 Foveated Block DCT 

The Foveated Block DCT may be implemented in C ++ 
(Watcom C/C + *) on an Intel Pentium® processor. The computations will 
be of the form typical in the literature, with the addition of the 
modifications for foveated imaging described in 2.1. Specifically, the 
THRESHOLD and QUANTIZE operations will be applied to each of DCT 
components (coefficients) computed in each block. The only difference is 
that I in equations (17)-(22) will now represent a particular DCT 
component (i.e., basis function) in a particular block, u, will represent the 
dominant spatial frequency of the component, and e will represent the 
eccentricity of the center of the particular block. Equation (20) will be 
utilized to determine those blocks for which each DCT component must 
be computed. 

4.5 FIS Perceptual Evaluation 

The current invention (running under Mode 1) has been 
evaluated for perceptual performance using a conventional 21 inch video 
display monitor. The images were 256 x 256, 8 bit gray scale images, 
having a 20° field of view, displayed with eye movement update rates of 
30 Hz, at a frame rate of 60 Hz, and with a half-resolution constant (e^ 
of 2.5° . The images were selected to test the perception of a number of 
probable image types: a letter chart (evaluation of visual clarity in a 
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reading task), a face (evaluation of video telecommunication systems), and 
a natural environment scene (evaluation of cluttered, high detail images). 
User reports of the subjective quality of the displays were used in the 
evaluation. All subjects have reported smooth, accurate tracking and 
excellent overall functioning of the foveating algorithms. Upon steady 
fixation, most subjects noted that they were aware of the reduced 
resolution in the peripheral visual field, but that the effect was minimal 
and had only a small impact on the perceived quality of the image. 

In a more quantitative study, the inventors measured visual 
search performance (in Mode 1) for a 1.0° x 0.6° target in natural scenes, 
under three different conditions: (1) foveated imaging of 20° x20° images, 
(2) windowed viewing (4.9° x 4.9°) with the number of pixels equal to that 
of the foveated image, and (3) constant resolution viewing (20° x 20°) 
with the number of pixels equal to that of the foveated image. The results 
showed that the search speed and accuracy was much greater for the 
foveated imaging. These results illustrate the potential value of the 
foveated imaging system in important real-world applications where 
transmission bandwidth is limited. 

4.6 Computational Performance and Bandwidth Reduction 

Use of the Foveated Imaging System has demonstrated that 
significant bandwidth reduction can be achieved, while still maintaining 
access to high detail at any point of a given image. In Mode 1, using 
19 x 22.5 cm. images, at a viewing distance of 30 cm, with a half 
resolution constant (e^ of 2.5° and a fovea! SuperPixel size of 1 pixel, the 
inventors are able to achieve bandwidth reductions of 97% (a factor of 32) 
for 644 x 768 images, on a 200 mHz Pentium® computer. For these 
systems parameters and images, the eye-movement update rate for thee 
display exceeded 12 Hz for 24-bit color images, and exceeded 17 Hz for 8- 
bit gray scale images (the refresh rate of the display was 60 Hz). These 
bandwidth reductions are independent of the particular images and are 
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obtained without any form of secondary compression. A computer coding 
used in a preferred embodiment of the present invention is included with 
this application as an Appendix I. 

Secondary image compression schemes, such as thresholding, 
quantization, and image-frame differencing, can be used in conjunction 
with the foveated imaging system. The effects of such secondary 
compression are essentially multiplicative; for example, a factor of 20 in 
secondary compression would yield an overall compression factor of 640. 

Non-foveated Laplacian and wavelet pyramids often produce 
quite acceptable 8-bit gray images with bit rates of 0.1— 0.5 bits/pixel (Burt 
& Adelson, 1984; Said & Pearlman, 1996). The Foveated Laplacian and 
Foveated Wavelet Pyramids should produce quite acceptable foveated 
images at bit rates many times lower. 

It will be obvious to those having skill in the art that many 
changes may be made in the above-described details of a preferred 
embodiment of the present invention without departing from the 
underlying principles thereof. The scope of the present invention should, 
therefore, be determined only by the following claims. 
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APPENDIX 1 



CURSORCXX: 
#include "cursor.h n 

void Cursor: :UpdateX (int x) 
{ 

_x += x; 

if (_x > (int) (_max_x - _width / 2 - 1)) 

_x = _max_x - width / 2 - 1; 
elaeif (_x < (int) _width / 2) 
x = _width / 2; 

} 

void CurBor::UpdateY (int y) 
{ 

_y + =jr. 

if (_y > (int) (_max _y - _width / 2 - 1)) 

_y = _max_y - _width / 2 - 1; 
else if (_jr < (int) _width / 2) 

_y = width / 2; 

> 

// Turn off the following two warnings: 

// W604: must look ahead to determine whether construct 

// is a declaration/type or an expression 

// W594: construct resolved as a declaration/type 

#pragma warning 604 9 
^(tpr agnftfl w a rning 594 9 

void Cursor::Draw (void (*LineDraw) (unsigned xl, unsigned yl, unsigned x2, unsigned y2, unsigned long color)) 
{ 

// + 

LineDraw (_x + _off set_x - _width / 2, 
_y + _offeet_y, 
_x + _oflset_x - _width / 4, 
__y + _offset _y, 
_colorl); 

LineDraw (_x + of&etx - _width / 2, 
_y + _oflset^y + 1, 
_x + jofiset_x - _wxdth / 4, 
_y + jofiset^y +1, 
_colorl); 

LineDraw (_x + oflsetx + _width / 2, 
_y + joflset _y, 
_x + offsets + _width / 4, 
__y + _of3set__y, 
_colorl); 

LineDraw (_x + _offset_x + width / 2, 
_y + _o££set_y + 1, 
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_x + ofisetx + width / 4, 
_y + offeetjr + 1, 
_colorl); 

LineDraw (_x + _oflflet_x, 
_y + ofEset^y - _width / 2, 
_x + _offset_x, 

jf + ^offeetjr - _width / 4 + 1, 
color 1); 

LineDraw (_x + joffsetx + 1, 
jf + j>fifeetjr - _width / 2, 
_x + _offeetx + 1, 
_y + _of&etjyr - _width / 4 + 1, 
_colorl); 

LineDraw (_x + _ofTset_x, 

_y + offset^ + _width / 2, 

_x + _of&et_x, 

_y + _of&etjr + _width / 4, 

jcolorl); 
LineDraw (_x + _of!aet_x + 1, 

_y + oflset^y + _width / 2, 

_x + ofifeetx + 1, 

_y + _of&etjr + width / 4, 

jcolorl); 

//x 

LineDraw (_x + _offset_x - _width / 2, 
_y + _of&et_jr - _width / 2, 
_x + jofifeet_x - __width / 4, 
_y + offeet^y - _width / 4, 
_color2); 

LineDraw (_x + oflfeetx + width / 2, 
_y + _ofiset_y + _width / 2, 
_x + _of!set_x + _width / 4, 
_y + _oSset_y + width / 4, 
_color2); 

LineDraw (_x + _offeet__x + _width / 2, 
_y + _offset_jr - _width / 2, 
_x + _ofifiet_x + _width / 4, 
_y + _of&et_y - _width / 4, 
_color2); 

LineDraw {jl + offset s - width / 2, 
_y + _off8et_y + _width / 2, 
_x + _o£feet_x - _width / 4, 
_jr + _oflfeet_y + _width / 4, 
_co!or2); 



FOVEATOR.CXX: 
#include <assert.h> 
#include <math.h> 
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#include "foveator.h" 

Foveator::Foveator () 
{ 

Jialfjresolutum =5 0.0; 
^distance = 0; 
j3ixels_per_cm = 0; 
_wO = 0; 

_x_resolution = 0; 
_y_resolution == 0; 
_max_super_pixels = 0; 
_8uper__pixel_index = 0; 
superjixels = 0; 

} 

Foveator: : -Foveator () 
{ 

if (_super_pixels) 

deleted _Buper_pixelfl; 

} 

void Foveator: rSetScreenResolutdon (unsigned x_resolution, unsigned ^resolution) 
{ 

_x_resolution = xresolution; 
^y^resolution = y_resolution; 

if (__superj>ixels) 

delete [] _super_pixels; 

// The number of superpixels allocated is based upon the image size. 

_max_super_pixe3fl = _x_resolution * _y_resolution; 
_euperjpixels = new SuperPixel [jnax_superj>ixels]; 

if ( ! _super_pixels) 

throw To veator : :AllocateSuperPixels( ) : memory allocation error"; 

}; 

SuperPixel *Foveator::Foveate (unsigned center x, unsigned center jr, unsigned &super _pixel_count) 
{ 

if (center_x > _x_resolution) 

throw "Foveator: :Foveate(): X center value is out of range"; 

if (center_y > _y_resolution) 

throw "Foveator: :Foveate(): Y center value is out of range"; 

if (! jxjresolution 8 l^y^resolution) 

throw "Fovaator::Foveate(): a non-zero X and Y resolution must be specified"; , t o-~ 

if (!_distance) 

throw "Foveator: :Foveate(): a non-zero value for distance be specified"; 
const double PI = 2 * asin (1.0); 
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unsigned max_resolution = (_x_resolution < _y_resolution) ? _y_resolution : _x_resolution; 

// Compute the super pixels 

_8uper_pixel_index = 0; 
unsigned pixelsize; 

float half_res_pixels = Jialf jresolution * tan (2 * PI / 360) * 
^distance * _pixels_per_cm ; 

for (unsigned x = 0; x < max resolution; x += pixeljaize) 
{ 

pixel size = (unsigned) ((_w0) * (1.0 + x / halfresj>ixels) + 0.5); 

if (pixel_size < 1) 
pixelsize = 1; 

// // Round to nearest power of 2 

// 

// unsigned log_2 = (unsigned) (log (pixel jrize) / log (2.0)); 
// 

// pixel size = 1 < < log_2; 
unsigned pixeljieight; 

for (unsigned y = 0; y < = x; y + = pixelheight) 
{ 

pixelheight = pixelsize; 

// If the next superpixel is going to be truncated, 
// decide whether or not we should connect it to this 
// super pixel 

if (y < x && y + pixeljieight + pixel Jieight / 2 > x) 

pixel height = x - y; 
else if (y < x && y + pixeljieight > x) 

pixeljieight = x - y; 

AddSuperPixels (center_x, center jr, x, y, pixeljrize, pixel height); 

> 

} 

super_pixel_count = _super jMxelJndex; 
return jBuper__pixels; 



void Foveator: lAddSuperPixels (unsigned center_x, unsigned center^y, „ 

unsigned x, unsigned y, 
unsigned width, unsigned height) 

{ 

int xl = x; 
int yl = y; 
int x2 = x + width; 
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int y2 = y + height; 

// Rotate about center counter clockwise starting at 3:00 

AddSuperPixel (centerjc + xl, center_y + yl, width, height); 
AddSuperPixel (center_x + yl, center _y - x2, height, width); 
AddSuperPixel (centerjc - x2, center _y - y2, width, height); 
AddSuperPixel (centerjc - y2, center __y + xl, height, width); 

// Only add superpixels that are on the diagonal one time 

if (x ==y) 
return; 

// Mirror about the diagonal 
xl=y, 

x2 = y -r height; 
y2 = x + width; 

// Rotate as before, switching width and height 

AddSuperPixel (centerjc + xl, center_y + yl, height, width); 
AddSuperPixel (centerjc + yl, center^y - x2, width, height); 
AddSuperPixel (centerjc - x2, center^y - y2, height, width); 
AddSuperPixel (centerjc - y2, center_y + xl, width, height); 



void Foveator : :AddSuperPixel (int x, int y, unsigned width, unsigned height) 
{ 

int left = x; 

int bottom = y t 

int right = x + width - 1; 

int top = y + height - 1; 

// Check to see if it is within screen limits 

if (top < 0 | right < 0) 
return; 

if (left > (int) ( jcjresolution - 1) | bottom > (int) (_ - y_resolution - 1)) 
return; 

// Check to see if it overlaps a screen edge, and truncate if necessary 

if (left < 0) 

left = 0; , t ^.v. 

if (bottom < 0) 
bottom = 0; 

if (right > (int) (jcjresolution - 1)) 
right = (int) (jcjresolution - 1); 
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if (top > (int) (^resolution - 1)) 
top = (int) (^resolution -1); 

//Add it 

assert (_super_pixel_index < _max_8uperj>ixels); 

SuperPixel *super_jrixel = &_Buper_pixels[_superj>ixel_index] ; 

super_pixel->x = (unsigned short) (left); 
super_pixel->y = (unsigned short) (bottom); 
super_pixel- > width = (unsigned short) (right - left + 1); 
super jpixel- > height = (unsigned short) (top - bottom + 1); 

++ super_pixel_index; 

} 

INTEGRAT.CXX: 
#include < assert. h> 
#include <string.h> 

#include "integrat.h" 
#include M tga.h M 
#include "yuv.h" 

void Integrate (SuperPixel *super_pixels, 
unsigned super_pixel_count, 
void *source_image, 
unsigned image width, 
Palette *palette, 
unsigned tga image type code) 

{ 

for (unsigned super_pixel_index - 0; superj>ixel_index < super__pixel_count; super_pixel_index + + ) 
{ 

SuperPixel *super_pixel = &super_pixels [super_pixel_index] ; 

unsigned super_pixel_x = superj>ixel->x; 
unsigned super_pixel_y = super_pixel->y; 
unsigned euper_pixel_width = super_pixel-> width; 
unsigned super_pixel_height = superjpixel-> height; 

switch (tga image type_code) 
< 

case TGA_COLOR_MAPPED : 
{ 

RGB rgbtemp; 
RGB rgbjivg; 

rgb_avg.r ■= 0; 
rg*>_avg.g = 0; 
rgb_avg.b = 0; 

for (unsigned y index = 0; yindex < super_pixel_height; y_index++) 
( 
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unsigned y_ofiset = (y_index + super_pixel_y) * image_width + super_pixel_x; 

for (unsigned x index = 0; xindex < superj>ixel_width; x_index++) 
{ 

rgb_temp = palette- > Get (((char *) source__image) [y_offset + x_index]); 

rgb_avg.r + = rgb_temp.r; 
rgb_avg.g + = rgb_temp.g; 
rgb_avg.b += rgbtemp.b; 

> 

} 

rgb_avg.r /= (6uper_pixel_height * superjpixel_width); 
rgb_avg.g /= (super jixeljieight * super_pixel_width); 
rgb_avg.b /= (super_pixel_height * super_pixel_width); 

int color = YUVFindClosestRGB (rgbjivg, ♦palette); 

assert (color < 256 && color >= 0); 

8Uper_pixel-> color = color; 

} 

break; 

case TGA_TRUE_COLOR: 
{ 

RGB rgb avg; 
rgb_avg.r = 0; 
rgb_avg.g = 0; 
rgb_avg.b = 0; 

unsigned long color, 

for (unsigned y index = 0; yindex < super_pixel_height; y__index++) 
{ 

unsigned y_offset = (yjndex + super_pixel^y) * imagejwidth + super_pixel_x; 

for (unsigned x index = 0; x_index < super__pixel_width; x_index++) 

{ . 

color = *(((long *) source_image) + y offset + x index); 

rgbjivg.r += (color & GxOOFFOOOO) >> 16; 
rgbjavg.g + = (color & OxOOOOFFOO) >> 8; 
rgb_avg.b += color & OxOOOOOOFF; 

} 

} 

rgbavg.r /= (superj)ixel_height * super_pixel jwidth) ; 
rgb_avg.g /= (superj>ixel_height * superj>ixel_width); 
rgp_avg.b /= (euper_pixel_height * super jtixel_width); 

color = rgb_avg.r << 16; 
color += rgb avg.g << 8; 
color + = rgbavg.b; 
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assert (color < 0x01000000); 
super_pixel- > color = color; 

} 

break; 

case TGA_GREY_SCALE : 
{ 

int color = 0; 

for (unsigned y index = 0; y_index < super_pixel_height; y_index++) 
{ 

unsigned y_offset = (y_index + super jpixel_y) ♦ imags^width + superj>ixel_x; 

for (unsigned x_index = 0; x index < super_pixel_width; xjndex++) 
color + = ((char *) source image) [y offset + x_index]; 

) 

color /= (super_pixel_height * super_pixel_width); 
assert (color < 256 && color >= 0); 
super_pixel- > color = color, 

> 

break; 
default; 

throw "Invalid type code"; 

} 

} 

} 

INTERPOL.CXX: 
#include <stringJi> 

#include "interpoLh" 
#include "tgaJi" 

void Interpolate (SuperPixel *super_pixels, 
unsigned super_pixel_count, 
char *dest_image, 
unsigned image_width, 
unsigned tgajmage type_code) 

< 

for (unsigned super_pixe]_index = 0; superjrixeljndex < super_pixel_count; super - _pixel_index++) 
{ 

SuperPixel *superj>ixel = &super_pixels[super_pixel_index] ; 

.-wr-»« . unsigned 8uper - _pixel_x = super_pixel->x; 
unsigned super_pixel_y = super_pixel->y; 
unsigned super_pixel_width = super_pixel-> width; 
unsigned super_pixel_height = super_pixel-> height; 
unsigned long super_pixel_color = super_pixel- > color, 
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switch (tgajmage Jype_code) 
{ 

case TGA_COLOR_MAPPED: 
{ 

for (unsigned yindex = 0; y_index < super_pixel_height; y_index++) 
{ 

unsigned offset = (y_index + super jMxel_y) * image_width + 8uper_pixel_x; 
memset ((void *) ( &dest_image[offset] ), super_pixel_color, super _pixel_width); 

} 

} 

break; 

case TGA_TRUE_COLOR: 
{ 

for (unsigned yindex = 0; y_index < superjpixelheight; y_index++) 
{ 

unsigned offset = (y index + super_pixel_y) * image width + super_pixel_x; 

for (unsigned xindex = 0; xindex < super_pixel_width; x_index++) 
♦(((long *) destjmage) + offset + xjndex) = 8uper_pixel_color f 

} 

} 

break; 

case TGAGREYJSCALE : 
{ 

for (unsigned y index = 0; y index < super_pixel_height; y_index++) 
< 

unsigned x_offset = (y_index + super_pixel_y) * image_width + super_pixel_x; 
memset ((void *) (&destJmageIx_offset]), superj>ixel_color, super_pixel_width); 

> 

} 

break; 
default: 

throw Invalid type code"; 

) 



LEVEL.CXX: 
#include <string.h> 

#include u assert.h? 
^include leveLh" 

void Reduce2x2 (Level *a, Level *b) 
{ 

// Average blocks in a and store them in b 
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unsigned width = a- > width; 
unsigned height = a- > height; 
char *image_a = a- > image; 
char * image b = b-> image; 

for (unsigned y index = 0; y_index < height - 1; y index +=2) 
{ 

for (unsigned x_index = 0; xindex < width - 1; xjndex +=2) 
{ 

assert (yjndex * width + x_index < a- > width * a- > height); 
assert (y_index * width + x_index + 1 < a- > width * a-> height); 
assert ((y_index + 1) * width + xjndex < a- > width * a->height); 
assert ((y_index + 1) * width + x_index + 1 < a- > width * a->height); 

int avg = image_a(y_index * width + x_index]; 

avg + = image_a[y_index * width + xindex + 1]; 
avg + = image_a[(y_index + 1) * width + xjndex]; 
ave += image_a[(y_index + 1) * width + x_index + 1]; 

avg/= 4; 

assert (y_index / 2 * width / 2 + x index / 2 < b->width * b->height); 
image bfy_index / 2 * width / 2 + x index / 2] = (char) (avg); 

} 

> 

} 

void Expand2x2 (Level *a, Level *b, Threshold * threshold) 
{ 

// Interpolate pixels in 'a' and store them in V 

// 'b' is stored as a sequential block, with the top left 
// pixel starting in b's image buffer at index 0. 

unsigned xl; 
unsigned yl; 
unsigned x2; 
unsigned y2; 

if (threshold) 
{ 

// Only interpolate in V over the region from (xl, yl) to (x2, y2) 

xl = threshold->xl; 
yl = threshold- >yl, 
x2 = threshold- >x2; 
y2 = threshold->y2; 

} 

else 
{ 

// Do the entire block 
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xl = 0; 
yl = 0; 

x2 = b-> width; 
y2 = b->height; 

} 

unsigned b_width = x2 - xl; 
unsigned bheight = y2 - yl; 

// First do all of the inside 2x2 blocks 

for (unsigned y_index = 1; yjndex < b height - 1; yjndex +=2) 
{ 

for (unsigned xjndex = 1; xjndex < b_width - 1; xjndex += 2) 
{ 

unsigned a jndex = (yl + yjndex) / 2 * a- > width + (xl + xjndex) / 2; 

assert (a index < a- > width * a- > height); 

char *pl = &a- > image[a jndex) ; 

char vl = (char) ((*pl)/2); 

char v2 = (char) ((*(pl + D) / 2); 

char v3 = (char) ((*(pl + a->width)) / 2); 

char v4 = (char) ((*(pl + a- > width + 1)) / 2); 

unsigned b jndex = yjndex * b_width + xjndex; 

assert (b jndex + bwidth + 1 < b-> width * b->height); 

char *p2 = &b- > image [b jndex] ; 

♦p2 a (char) (vl + (v2 + v3) / 2); 

♦(p2 + 1) = (char) (v2 + (vl + v4) / 2); 
*(p2 + bwidth) = (char) (v3 + (vl + v4) / 2); 
*(p2 + b width + 1) = (char) (v4 + (v2 + v3) /2); 

} 

} 

// Now do the 1 pixel wide edge around the whole thing 

for (unsigned xjndex = 0; x index < b width; xjndex++) 
{ 

// Top and bottom 

assert (xjndex < b-> width * b-> height); 

assert ((bheight - 1) * b_width + xjndex < b->width * b->height); 

b-> image [xindex] = b- > image [b_wid th + xjndex]; 
b- > image [(b Jieight - 1) * b width + xjndex] = 
b->image[(b Jieight - 2) * b_width + xjndex]; 

} 

for (yjndex = 0; yjndex < b Jieight; y_index++) 
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// Left and right 

assert (y_index * b_width < b-> width * b-> height); 

assert (y_index * b'width + (b_width - 1) < b->width * b->height); 

b->image(y - index * b_width] = b- > image |y_index * b_width + 1]; 
b- > image [y_index ♦ b_width + (bjwidth - 1)] = 
b- > image [y_index * b_width + (b width - 2)]; 

) 

> 

void Reduce3x3 (Level *a, Level *b) 
{ 

Reduce2x2 (a, b); 

} 

void Expand3x3 (Level *a, Level *b, Threshold ♦threshold) 
{ 

Expand2x2 (a, b, threshold); 

} 

void Subtract (Level *a, Level *b, Threshold *threshold) 
{ 

//b = a-b 

unsigned xl; 
unsigned yl; 
unsigned x2; 
unsigned y2; 

if (threshold) 
{ 

xl = threshold->xl; 
yl = threshold- >yl; 
x2 = threshold->x2; 
y2 = threahold->y2; 

} 

else 
{ 

xl = 0; 

yi = o ; 

x2 = a->width; 
y2 = a=>height; 

} 

// The difference is stored in V sequentially 

unsigned b index — 0; 

for (unsigned y = yl; y < y2; y++) 
{ 

for (unsigned x = xl; x < x2; x++) 
{ 
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assert (bindex < b->width * b->heig£t); 

assert (y * a->width + x < a->width * a->height); 

b- > image[b_index) = 

(char) (a->imagely * a->width + x] - b->image[b_index]); 

+ +b_index; 

} 

} 

if (threshold) 
{ 

unsigned b_width = x2 - xl; 
unsigned b_height = y2 - yl; 

for (unsigned x_jndex = 0; x_index < bwidth; x_index++) 
{ 

// Top and bottom 

assert (x_index < b-> width * b-> height); 

assert ((b_index - 1) * b_width + xjndex < b-> width * b-> height); 
b->image[x_index] - 0; 

b- > image[(b_height - 1) * b width + xjndex] = 0; 

> 

for (unsigned yjndex = 0; y index < bheight; y_index++) 
{ 

// Left and right 

assert (yjndex * b_width < b->width * b->height); 

assert (yjndex * b_width + (bwidth - 1) < b-> width * b->height); 

b- > image [yjndex * b_widthl = 0; 

b- > image [yjndex * b width + (b_width - 1)] = 0; 

} 

} 

} 

void Add (Level *a, Level *b, Threshold *threshold) 
{ 

//b b a + b 

unsigned xl; 
unsigned yl; 
unsigned x2; 
unsigned y2; 

if (threshold) 
{ 

xl - threshold->xl; 
yl - threshold- >yl; 
x2 - threshold->x2; 
y2 - threshold->y2; 
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) 

else 
{ 

xl = 0; 
yl = 0; 

x2 = b-> width; 
y2 = b->height; 

} 

// The contents of 'a' are stored sequentially 

unsigned aindex = 0; 

for (unsigned y = yl; y < y2; y++) 
{ 

for (unsigned x = xl; x < x2; x++) 
{ 

assert (a_index < a- > width * a-> height); 

assert (y * b-> width + x < b->width * b-> height); 

b-> image [y * b-> width + x] += a- > image [aindex] ; 

+ +a_index; 

> 

} 

} 

void Copy (Level *a, Level *b) 
{ 

b-> width = a- > width; 
b-> height = a- > height; 

memcpy (b-> image, a- > image, a- > width * a- > height); 

) 

MOUSE.CXX: 
#include <doah> 

#include "mouse.h" 

int Mouselnit () 
{ 

REGS regB; 

// SW Mouse reset 

regs.w.ax = 0x21; 

int386 (0x33, &regB, &regB); 

// Mouse init 

regs.w.ax = 0x00; 
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int386 (0x33, &regs, &regs); 

return (regs.w.ax == 0x21); // return 0 on success 

} 

unsigned short MouseButtonStatus () 
{ 

REGS regs; 
regB.w.ax = 0x03; 
int386 (0x33, &regs, &regB); 
return regB.w.bx; 

} 

unsigned long MousePosition () 
{ 

REGS regs; 

regs.w.ax = 0x03; 

int386 (0x33, &regs f &regs); 

// Return column in upper word and row in lower word 
return ((unsigned long) regs.w.cx) << 16 | regs.w.dx; 

> 

unsigned long MouseMotion () 
{ 

REGS regs; 

regs.w.ax = OxOB; 

int386 (0x33, &regs, &regs); 

// This returns the number of mickeys moved since the last call. 

// A mickey is the smallest unit a mouse can move. 

// The upper word contains movement in the x direction, and 
// the lower word contains movement in the y direction. 

// Note that the motion can be a negative value. 

return ((unsigned long) regs.w.cx) < < 16 | regs.w.dx; 

} 

PALETTE.CXX: 
#include <math.h> 

#include "palette.h" 

Palette:. Palette (unsigned size, unsigned bits_per_primary) 
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{ 

_size = size; 

_bits_per_primary = bita_per_primary f 
_rg)t>_table = new RGB [size]; 

if (!_rgb_table) 

throw Talette::Palette(): Memory allocation error n ; 

LoadGreyScales 0; 

} 

Palette:: -Palette () 
{ 

delete _rgb_table; 

} 

void Palette: :Set (unsigned index, unsigned greyjscale) 
{ 

if (index >= jaze) 

throw "Palette-SetO: Color index out of range"; 

_rgb_table[index] . r =5 
rgbtablefindex] .g = 
j^tablefindexl.b =5 greyjscale; 

} 

void Palette:: Set (unsigned index, unsigned r, unsigned g, unsigned b) 
{ 

if (index > = jsize) 

throw "PaIette::Set(): Color index out of range"; 

_rgb_table[index] .r = r, 
_rgb_table [index] .g = g; 
_rgb_table[index].b = b; 

} 

void Palette:: Set (unsigned index, RGB &rgb) 
{ 

if (index > = jsize) 

throw "Palette::Set(): Color index out of range"; 

_rgb_table [index], r = rgb.r; 
_rgb_table [index] . g = rgb.g; 
_rgb_table [index] .b = rgb.b; 

} 

RGB &Palette::Get (unsigned index) 
{ • . - 

if (index > = jsize) 

throw "Palette ::Get(): Color index out of range"; 

return rgptable [index] ; 

} 
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void Palette::LoadGreyScalea 0 

for (unsigned index = 0; index <_size; index ++) 
{ 

unsigned color = (unsigned long) index * (1 < < _bitsj>er j>nmary) / _size; 

_rgb_table [index] .r = 
_rgb_table [index] . g = 
_rgbjable[index].b = color; 

> 

} 

PROMPT.CXX: 
#include <stdio.h> 
#include <stdlib.h> 
#include <string.h> 

#include "prompt.h n 

void GetString (char *string) 
{ 

// Read a string from stdin, but don't overwrite 
// argument if nothing is entered. 

char buffer [80]; 

fgets (buffer, 80, stdin); 

if (strlen (buffer)) 

buffer[strlen(buffer) - 1] = '\0>; // delete <CR> 

if (strlen (buffer) > 0) 
strcpy (string, buffer); 

} 

void GetFloat (float &result) 
{ 

// Read a float from stdin, but don't overwrite 
// argument if nothing is entered. 

char buffer[80]; 

fgets (buffer, 80, stdin); 

float temp = atof (buffer); 

if (temp) 

result = temp; 

} 

void GetUnsigned (unsigned &result) 
{ 

// Read an unsign ed from stdin, but don't overwrite 
// argument if nothing is entered. 
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char buffer [80]; 

fgets (buffer, 80, stdin); 

unsigned nTemp = atoi (buffer); 

if (nTemp) 

result = nTemp; 

} 

PTIMER.CXX: 
#include "ptimer.h" 

static unsigned long start_high, start low; 
static unsigned long stop_high, stop Jo w; 

extern void asmjBtart_timer (); 
extern void asm_stop_timer (); 

// 0F3 lh is the Pentium ReaD Time Stamp Counter (KDTSC) Instruction 

#pragma aux asm start timer = \ 

"dbOFh" \ 

°db 3 lh" \ 

"mov start_high,edx" \ 

"move Btart_low,eax" \ 

modify [edx eax]; 
#pragma aux asm stop_timer = \ 

"dbOFh" ~ \ 

"db 3 lh" \ 

"move stop_high,edx" \ 

"move stop_low,eax n \ 

modify [edx eax] ; 

PentiumTimer : iPentiumTimer () 
started_flag = 0; 



void PentiumTimer: :Start () 
stertedjlag = 1; 
asm_start_timer 0; 



void PentiumTimer : :Stop () 
started Jlag = 0; 
asm_stop_timer (); 

void PentiumTimer ::Elapsed (unsigned long &high, u nsigne d long &low) 
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if (started Jlag) 

asmjstopjdmer (); 

High = stop high - start_high; 

if (stop Jow > startjow) 

low = stop_low - startjow; 
else 
{ 

// If the low part has looped, borrow from the high part 

low = OxFFFFFFFF + (stopjow - start Jow); 
high-= 1; 

) 



unsigned long PentiumTimer : :HighReg () 
asm^stop Jimer (); 
return stop high; 



unsigned long Pentium Timer: :LowReg 0 

asm_stop timer (); 

return stopjow; 

PYRAMID.CXX: 
#indude <assert.h> 
#indude <math.h> 
#indude <stringJi> 

#include "pyramidLh" 

Pyramid: : Pyramid (unsigned basewidth, 
unsigned basejieight, 
unsigned levels, 
float contrast_threshold, 
float halfresolution, 
float dominant_freqO, 
float image_distance, 
float pixel_per_cm) 

{ 

if (!(base_width >> Qevels - 1))) 

throw "Pyrainid: :Pyramid(): Too many levels specified for this width"; 

if (!(base_height » Qevels - 1))) 

throw pyramid: :Pyramid(): Too many levels specified for this height*; 

basewidth = base_width; 
base height = basejieight; 
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if (levels < 2) 

throw "Pyramid: :Pyramid(): You must specify more than 1 level"; 
levels = levels; 

_contrast_threshold = contrast_threshold; 
_half_resolution = half_resolution; 
_dominant_freqO = dominant_freqO; 
image di sta n ce = image distance; 
_pixels_per_cm = pixels_per_cm; 

Jrigjevele = 0; 
_small_levels = 0; 

// Level N does not get thresholded 

^thresholds = new Threshold [levels - 1]; 

if (^thresholds) 

throw "Pyramid::PyramidO: memory allocation error 1 '; 

> 

Pyramid:: "-Pyramid () 
< 

delete U thresholds; 

> 

void Pyramid: :ComputeThresholds (unsigned x, unsigned y) 
{ 

if (_big_levels == 0) 

throw "Pyramid : :Threshold() : levels have not been allocated yet"; 

// The last level does not get thresholded 

for (unsigned level = 0; level < JevelB - 1; level ++) 
{ 

int max_ecc = MaxEcc (level) >> level; 
if (maxecc < 2) 

throw "Pyramid: :CJomputeThresholds(): An image has been thresholded beyond the lower limit"; 

int xl = (x >> level) - maxecc; 
int yl = (y >> level) - max ecc; 
int x2 = (x > > level) + max ecc; 
int y2 = (y > > level) + max_ecc; 

// Align thresholds on even boundaries to ensure that levels 
// will expand properly. Otherwise, a roundoff can cause two 

// levels to be off by one pixel v 

if (xl& 1) 
+ +xl; 

if (yl&l) 
++yl; 



51 



SUBSTITUTE SHEET (RULE 26) 



WO 98/33315 



PCT/US97/23737 



if (x2 & 1) 
if (y2& 1) 

-y2; 

// Clip the thresholded region on the image boundaries 

if (xl < 0) 

thresholds f level ] x 1 = 0; 

else 

_thresholds(level]jcl = xl; 

if (yl < 0) 

_thresholdsflevel] .yl = 0; 
else 

_thresholds[levelJ.yl = yl; 

if (x2 > Jtigjevels[level].width) 

thresholds! level] .x2 = Jbi^levelsflevel] .width; 

else 

_thresholdaflevBl] js2 = x2; 

if (y2 > Jbigjevelsflevell .height) 

_thresholds[levell .y2 = _big_levels [level] .height; 
else 

thresholds Re vel].y2 = y2; 

} 

} 

unsigned Pyramid: :MaxEcc (unsigned level) 
{ 

const float NEG_ALPHA = -0.005; 

float max_ecc = (halfresolution / (jlominantfreqO / (1 < < level))) * 
eqrt (log (^contrast^threshold) / NEG_ALPHA) - _half_resolution; 

const float PI = 2 * asin (1.0); 

max ecc = max_ecc * tan (2 * PI / 360) * _image_distance * jrixelsjperjan; 
return max ecc; 

} 

EncodePyramid::EncodePyramid (unsigned base_width, 

unsigned basejxeight, 

unsigned levels, 

float contrast threshold, 

float half jnesolution, 

float dominant JreqO, 

float image_distance, 

float pixels_per_cm) : 
Pyramid (basejwidth, 
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base height 
levels, 

contrastjhreshold, 
halfresolution, 
dominant freqO, 
image_distance, 
pixels_per cm) 

{ 
} 

EncodePyramid::-EncodePyramid () 
{ 

// Deallocate ail of the levels, but only if AllocateLevels was called 

if (!J>igJevels) 
return, 

// Don't deallocate the image in level 0 because this 
// was allocated externally. 

for (unsigned level = 1; level < Jevels; level++) 
{ 

if ( Jrigjevels [level J .image) 

delete Q Jrigjevelsflevd] .image; 

if (_small_le vels [level], image) 

delete [) small Jevelsflevel]. image; 

> 

delete Q Jrigjevels; 
delete [] small Jevels; 

} 

void EncodePyramid : : AllocateLevels (Level *basejevel) 
{ 

// Allocate the level structures 

Jrigjevels = new Level [Jevels]; 
small Jevels = new Level [Jevels]; 

if (! Jrigjevels | !_small Jevels) 

throw "EncodePyramid: :AllocateLe velsO : memory allocation error"; 

// Allocate all the image buffers within the levels and set their 
// correct sizes 

// When encoding, the lowest big level's image is not allocated, 
// instead, the image is external. 

J}igjevels[0].image = base level- > image; 
Jrigjevels[0].width = base Jevel-> width; 
J>igJevels[OJ.height = base Je vel- > height; 

for (unsigned level = 1; level < Jevels; level++) 
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{ 

_big_levels [level] .width = _base_width >> level; 
_big_levels[level] .height = _base_height >> level; 

Jbig^levels [level] .image = 

new char [JrigJevelsQevel] .width * Jbig^levelsQevel] .height]; 

if (IJbigJeveMlevdl.image) 

throw "EncodePyramid: lAllocateLevelsO : memory allocation error"; 

} 

// When encoding, the highest small level is not used 

jamall_levels[Jevels - 1] .width = 0; 
_Bmall_levels[_levels - 1] .height = 0; 
jBmall_levelsl_levels - 1] .image = 0; 

for Oevel = 0; level < Jevels - 1; level++) 
{ 

smalllevelfl [level] .width = _base_width > > level; 
jamallJeveJs [level] .height = _base_height » level; 

// When encoding, the small image buffers can't be 
// bigger than the largest thresholded region. 

if (((MaxEcc (level) * 2) >> level) < _smalljevels[level] .width) 
jmiall_levels[level] .width = (MaxEcc aevel) * 2) >> level; 

if (((MaxEcc (level) * 2) >> level) < jmialljevels [level] .height) 
jBmall_levels [level] .height = (MaxEcc (level) * 2) » level; 

jamallJLevelsflevel]. image = 

new char [_small_le vels[level] . width * _amall_levels[level] .height] ; 

if (!_smalljevels [level] .image) 

throw "EncodePyramid:LAllocateLevels(): memory allocation error"; 

} 

> 

void EncodePyramid : : Reduce (unsigned level) 
{ 

// Reduce level to level + 1 

if (level >= levels - 1) 

throw "EncodePyramid: iReduceO: level is out of range"; 

::Reduce2x2 (&_big_levels[level] f &_big_levels [level + 1]); 

} " - 

void EiicodePyramid : :Expand (unsigned level) 
{ 

// Expand level to level - 1 
if (level > = Jevels) 
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throw "EncodePyramidrrExpandO: level is out of range"; 
if(level<l) 

throw 0 EncodePyramid::Expand(): level is out of range* 1 ; 
::Expand2x2 (& bigjevels[levell, &_small_levels [level - 1], &Jhresholds[level -1)* 

} 

void EncodePyramid "Subtract (unsigned level) 
{ 

// Subtract the big level from the small level and store the 
// result in the small level 

if (level > = Jevels - 1) 

throw "EncodePyramid: :Subtract(): level is out of range"; 

::Subtract (&JWgJevels[tevel], & small Jevelsflevel], & thresholds[levelJ); 

} 

Level * EncodePyramid : : GetLe vel (unsigned level) 
{ 

// Return small level 

if (level >= Jevels - 1) 

throw "EncodePyramidil^velO: level is out of range"; 

return &_small levels [level]; 

> 

Level *EncodeI^amid::GetLevelN 0 
{ 

// Return highest big level 

Level *level = &_big^levels[_levels - 1]; 

return level; 

} 

DecodePyramid: :DecodePyramid (unsigned base width, 

unsigned base_height, 

unsigned levels, 

float contrast J&reshold, 

float halfresolution, 

float dominant JreqO, 

float image distance, 

float pixels_per_cm) : 
Pyramid (base_width, 

baseheight, 

levels, 

contrast^threshold, 

halfjresolution, 

dominant_freqO, 

image_distance, 

pixels_per_cm) 
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{ 
} 

DecodePyMonid:: -DecodePyramid () 
{ 

// Deallocate all of the levels, but only if AllocateLevels was called 

if (! J>igjevels) 
return; 

// No small levels are used when decoding 

for (unsigned level = 0; level < Jevels; level++) 
delete D _bigjevels[level]image; 

delete [] Jrigjevels; 

} 

void Decode Pyramid:: AllocateLevels () 
{ 

// Allocate the level structures 
Jngjevels = new Level [ levels]; 
if (! Jrigjevels) 

throw °EncodePyramid:LAilocateLevels(): memory allocation error"; 

// When decoding, no small levels are used 

for (unsigned level = 0; level < levels; level++) 
< 

Jrigjevels [level] .width = J>ase_width > > level; 
Jbigjevelsflevel] .height = _base_height >> level; 

Jrigjevels [level] .image = 

new char [JrigJevelsPevel]. width * JrigJevelsDevel] .height]; 

if (! Jrigjevels[level] .image) 

throw "DecodePyranud::AUocateLevels(): memory allocation error"; 

} 

> 

void DecodePyramid: :Expand (unsigned level) 
{ 

// Expand level to level - 1 

if (level >= Jevels) 

throw N DecodePyramid::Ezpand(): level is out of range"; 

if (level == 0) 

throw "DecodePyramid: :Expand(): level is out of range"; 

:£zpand2z2 (&JrigJevels[level], & bigjevels[level - 11); 

> 
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void DecodePyramidiLAdd (Level *differenced_level t unsigned level) 
{ 

// Add at the differenced level to the pyramid at the 
// specified level and store it in the big level 

if Gevel >= Jevels) 

throw "DecodePyramid::AddO: level is out of range"; 

::Add (differenced level, & bigjevelsflevel], & thresholds[level]); 

} 

Level 'Decode Pyramid: :GetLevel (unsigned level) 
{ 

// Return big level 

if (level > = Jevels - 1) 

throw T)ecodePyramid::GetLevel(): level is out of range"; 

return &_big_levels [level] ; 

} 

void DecodePyramid::SetLevelN (Level *level) 
{ 

// Set the topmost level of the pyramid 

if (level- > width != _big_levels[_levels - 1] .width) 

throw "DecodePyramid::SetLevelN(): width is incorrect 0 ; 

if (level- > height != _bigjevels[jevels - 1]. height) 

throw "DecodePyramid::SetLevelN(): height is incorrect"; 

memcpy (_big_levels[_levels - 1] .image, level- > image, 
level- > width * level-> height); 

} 

TGA.CXX: 
#include <stdio.h> 
#include <string.h> 

#include "tga.h" 

TGA;:TGA () 
{ 

image = 0; 
_width = 0; 
_height = 0; 

memset (&_tga_header, 0, sireof (_tga ^header)); 
^palette = 0; 

> 

TGA::-TGA () 
{ 

if ( image) 

delete □ jmage; 
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if (_palette) 

delete jpalette; 

} 

int TGA::Load (char *filename) 
{ 

FILE *tga_file = fopen (filename, W); 

if (!tga_file) 
return -1; 

Jgajieaderid Jieldjength = (unsigned char) (fgetc (tgajile)); 
_tga_header.color_map type = (unsigned char) (fgetc (tga_file)); 
_tga_header.image type code = (unsigned char) (fgetc (tgafile)); 

_tga_header.color_map_origin = (unsigned short) ((fgetc (tga_file) & OxFF) | (fgetc (tga_file) <<8)); 
_tga - header.color - map_length = (unsigned short) ((fgetc (tgajile) & OxFF) | (fgetc (tgajile) <<8)); 
_tga_header.color_map_entry_size = (unsigned char) (fgetc (tgajile)); 
JgaJieaderjt_origin = (unsigned short) ((fgetc (tga_file) & OxFF) | (fgetc (tgajile) <<8)); 
_tga_header.y_origin = (unsigned short) ((fgetc <tga_file) & OxFF) | (fgetc (tgajile) <<8)); 
Jgajieader.width = (unsigned short) ((fgetc (tga_file) & OxFF) | (fgetc (tga_file) <<8)); 
Jgajieader.height = (unsigned short) ((fgetc (tgajile) & OxFF) | (fgetc (tgajile) <<8)); 
_tga_header.bits_per jrixel = (unsigned char) (fgetc (tga file)); 
_tga head^.iirage descriptor = (unsigned char) (fgetc (tgajile)); 

for (unsigned index = 0; index <JgaJieader.id_fieldJength; index++) 
fgetc (tga_file); 

switch ( JgaJiea^er.imagejype_code) 
{ 

case 0: 

throw "TGA::Load(): The image contains no data"; 

case 1: // Color-mapped RGB 
case 2: // Indexed RGB 
case 3: // Grayscale 
break; 

case 9: 
case 10: 
case 11: 

throw TGA::Load(): Compressed TGA files are not supported"; 
default: 

throw "TGA::Load(): Unknown TGA file format is not supported"; 

} 

if ( tga header.color_map type) 

i , 

if (_tga_header.colorjrnap_entry_size != 24) 

throw "TGA::LoadO: For indexed RGB images, only 24 bit palettes are supported"; 

if (.palette) 

delete _palette; 
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_palette = new Palette (_tga_header.color__mapjength); 

if (!_palette) 

throw "TGA::LoadO: Memory allocation error"; 

for (index = 0; index < _tga_header.color map_length; index* +) 
i 

unsigned nB = fgetc (tga_file); 
unsigned nG = fgetc (tga_file); 
unsigned nR = fgetc (tga_file); 

_palette->Set (index, nR, nG, nB); 

} 

} 

else 
< 

if ( jpalette) 

delete _palette; 

jpalette = 0; 

> 

// Check to make sure the header was read in OK 

if (feof (tga file) | ferror (tgajBle)) 
{ 

fclose (tga_file); 

throw "TGALoadO: Invalid TGA file"; 

} 

if ( tga header.image descriptor & INTERLEAVE MASK) 
{ 

fclose (tgajile); 

throw TGA::Load(): Interleaved images are not supported"; 

} 

if (_image) 

delete [) image; 

J mage = 0; 

(_tga Jieader.fcts_^r_pixel ==8) 

_image = new char [_tga_header. width * _tga_header. height]; 
else if (_tga_header.bits_per _pixel ==24) 
{ 

// If it's a 24 bit per pixel image, let's convert it to 32bpp 
// in memory so that we can do 32 bit image processing 

_tga_header.bits_perjpixei = 32; 

Jmage = new char [_tga_header.width * _tga_header .height * 4]; 
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> 

else 
{ 

fclose (tgajile); 

throw "TGA::Load(): Only 8 or 24 bits per pixel images are supported"; 

> 

if (! image) 
i 

fclose (tga_file); 

throw "Memory allocation error"; 

) 

if (_tga header.image descriptor & SCREENORIGIN MASK) 
{ 

// Image is stored from top left to bottom right 

if (_tga_header.bits_per _pixel = =8) 
{ 

for (index = 0; index < _tga_header. width * _tga_header .height; index* +) 
Jmagefindex] = (char) (fgetc (tgajile)); 

} 

else 
{ 

//32bpp 

for (index = 0; index < Jgajieader.width * _tga_header .height * 4; index +=4) 

unsigned long dwPixel; 

dwPixel = fgetc (tga_ffle); // R 
dwPixel + = fgetc (tga_file) < < 8; // G 
dwPixel + a fgetc (tga file) < < 16; // B 

(* (unsigned long *) (&_image [index] )) = dwPixel; 

} 

} 

} 

else 
{ 

// Image is stored from bottom left to top right 
(_tga_header.bits_per_pixel ==8) 

( 

for (unsigned nHeight = _tga_header.height; nHeight > 0 ; nHeight-) 
{ Z 

for (unsigned nWidth = 0; nWidth < __tga_header. width; nWidth++) 

Jinage[(nHeight - 1) * J^Jieader.width + nWidth) = (char) (fgetc (tga_ffle)); 

> 

else 
{ 
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//32bpp 

for (unsigned nHeight = tga_header.height; nHeight > 0 ; nHeight-) 
< 

for (unsigned index = 0; index < tga_header. width; index++) 
{ 

unsigned long dwPixel; 

dwPixel = fgetc (tga_file); // R 
dwPixel + = fgetc (tgajile) < < 8; // G 
dwPixel + = fgetc (tgajile) < < 16; // B 

(* (unsigned long *) (&Jmage[(nHeight - 1) * tga header. width * 4 + index * 4])) = dwPixel; 

> 

} 

} 

> 

// Check to make sure the image was read in OK 

if (feof (tgajile) | f error (tga file)) 
{ 

f close (tgajile); 

throw "Invalid TGA file"; 

} 

fcloee (tga_file); 

return 0; // return 0 on success 

} 

VESA.CXX: 
#include <string.h> 

#include "vesa.h" 

static SVdevCtx *device_context = 0; 
static int vesa_init Jlag = 0; 

int VESAInit (unsigned x, unsigned y, unsigned depth bits, int bLinearBuffer) 
{ 

device_context = SVJnit (true); // true = use VBE/AF if available 

if (!device_context | device_context->VBEVersion < 0x200) 
return -1; 

unsigned short *mode; 

for (mode = device_context-> modeList; *mode != QxFFFF; mode + + ) 
{ 

SV_modeInfo Modelnfo; 

if (SV jgetModelnfo (*mode, &ModeInfo) && 

ObLinearBufTer | (ModelnfoAttributes & svHaveLinearBufTer)) && 
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ModelnfcXResolution == x && 
Modelnfo.YResolution == y && 
ModeInfo.BitaPerPixel == depthbits) 
break; 

} 

if (*mode == OxFFFF) 
return -1; 

if (!SV_setMode (*mode | (unsigned short) (bLinearBuffer ? svLinearBuffer : 0), 
true, // 8BitDAC 

true, // virtual linear framebuffer 
D) // # of multibuffer buffers 
return -1; 

vesa_init_flag = 1; 

return 0; 

} 

void VESAClose () 
{ 

if (vesa_init_flag) 

SV restoreMode (); 

vesa_init_flag = 0; 
devicejcontext = 0; 

} 

const VESADeviceContext *VESAGetDeviceContext () 
{ 

return devicejcontext; 

} 

void VESALoad (Palette &palette) 
{ 

SVjjalette *sv_palette = new SV_palette [palette.Size ()]; 

if (!sv_palette) 

throw "VESALoadPaletteO: Memory allocation error"; 

// A VESA palette entry is always 8 bits per primary 

if (palette3itsPerPrimary () != 8) 

throw "VESALoadPaletteO: VESA palettes must have 8 bits per primary"; 



// Loading palette entries one at a time doesn't work with 
r the dos32\wc\svga library dated 05/05/96. Instead, you 
// must load them all at once. 

for (unsigned index = 0; index < palette. Size (); index++) 
{ 

RGB rgb; 
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try 

{ 

rgb = palette. Get (index); 

> 

catch (...) 
{ 

delete [] sv_palette; 
throw; 

} 

sv_palette [index]. red = (char) rgb.r, 
sv_palette[index] .green = (char) rgb.g; 
sv_palette[index] .blue = (char) rgb.b; 
sv_palette [index] .alpha = 0; 

} 

SV_BetPalette (0, palette.Size (), evjwdette, -1); 
delete [] ev_palette; 



void VESALine (unsigned xl, unsigned yl, unsigned x2> unsigned y2, unsigned long color) 
SVJine (xl, yl, x2, y2, color); 



void VESABeginLine () 

SVJbeginLine (); 

SV beginDirectAccess (); 



void VESAEndLine () 

SV endDirectAjccesa (); 
SV endLine (); 



void VESALineFast (unsigned xl, unsigned yl, unsigned x2, unsigned y2, unsigned long color) 
SVJineFast (xl, yl, x2, y2, color); 



void VKSAPoint (unsigned x, unsigned y, unsigned long color) 
SVjputPixel (x, y, color); 



void VESABeginPoint () 
SV_beginDirectAccess (); 
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void VESAEndPoint () 
SV endDirectAccess (); 



void VESAPointF ast (unsigned x, unsigned y, unsigned long color) 
SV_putPixelFast (x, y, color); 



void VESAText (char 'szText, unsigned x, unsigned y, unsigned long color) 
SV_writeText (x, y, szText, color); 



void VESAScreenBlt (char 'bitmap, unsigned long length) 

if (! devicecontext) 

throw "VESAScreenBlt: VESA not initialized"; 

if (! device.context- > linearAddr ) 

throw "VESAScreenBltO: Not available in banked address mode"; 

SV_beginDirectAccess (); 

memcpy ((char •) device context- > videoMem, bitmap, length); 
SV endDirectAccess (); 

} 

void VESABitBlt (void 'bitmap, unsigned width, unsigned height, 
unsigned bytes_per_pixel, unsigned x offset, unsigned y_ofifeet) 

{ 

if (!device_context) 

throw ^VESABitBlt: VESA not initialized"; 

if (Idevice context- > linearAddr) 

throw "VESABitBltO: Not available in banked address mode"; 

SV_beginDirectAccess (); 

switch (bytes_per_pixel) 
i 

case 1: 
{ 

for (unsigned y =* 0; y < height; y++) 

memcpy ((char *) device_context-> videoMem + 
yjrf&et <* (device_context- > maxx + 1) + 
y * (device_context- > maxx + 1) + 
xoffset, 

(char *) bitmap + y * width, 
width); 

break; 
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} 

case 4: 
{ 

for (unsigned y = 0; y < height; y++) 

memcpy ((long *) device_context-> videoMem + 
y_offset * (device_context->maxx + 1) + 
y * (devicecontext- > maxx + 1) + 
x_ofIset, 

(long *) bitmap + y * width, 
width * 4); 

break; 

} 

default: 

throw "VESABitBltO: Invalid number of bytes per pixel"; 



SV endDirectAccess (); 

} 

void VESAClear (unsigned long color) 
{ 

SVclear (color); 

> 

YUV.CXX: 
#include <assert.h> 
#include <stdlib.h> 

#include "yuv.h" 

const unsigned YUV_MAP_LENGTH = 1 << (YWIDTH + UWIDTH + VWIDTH); 
static unsigned yuv jnap[rTJV_MAP_LENGTH] ; 

void YUVMapInit (Palette &palette) 
{ 

for (unsigned index = 0; index < YUV_MAP_LENGTH; index++) 
yuv_map [index] = -0; 

for (index = 0; index < 256; index* +) 
{ 

RGB rgb_temp = palette.Get (index); 
YUVyuvtemp; 

RGBtoYUV (rgbtemp, yuv_temp); 

unsigned long packed^yuv = PackYUV (yuv_temp); ~ r« 

assert (packedjniv < YUV_MAP_LENGTH); 
yuv map [packed jrav] = index; 

} 

} 
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unsigned YUVFindClosestRGB (RGB &rgb, Palette &palette) 
// RGB is any 24 bit RGB value 
// The return value is an index into Palette 
YUVyuvl; 

// Convert the RGB to 24 bit YUV 

RGBtoYUV (rgb, yuvl); 

// Scale down the 24 bit YUV to an index 

unsigned long yuvjndex = PackYUV (yuvl); 

assert (yuvjndex < YTJV_MAP_LENGTH) ; 

// If the RGB index for this YUV has not been computed yet, 
// then compute it now 

if (yuv maplyuv index] == -0) 
{ 

unsigned dosest_rgb_index = 0; 
unsigned closest^yuvdifference = -0; 

assert (palette. Size ()); 

for (unsigned index = 0; index < palette.Size (); index* +) 
{ 

YUVyuv2; 

RGBtoYUV (palette.Get (index), yuv2); 

// Compare the difference between our original 24 bit 
// YUV to the 24 bit YUV that corresponds with this 
// RGB palette entry. 

unsigned difference = YUVDifference (yuvl, yuv2); 

if (difference < dosest_yuv_difference) 
{ 

dosest^yuv^difference = difference; 
closest_rgb_index = index; 

} 

} 

yuvjnap (yuvjndex] = dosestjrgb_index; 

} .... • 

return yuv_niap[yuvjmdex]; 
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void RGBtoYUV (RGB &rgb, YUV &yuv) 
{ 

yuv.y = rgb.r * 0.299 + rgbg • 0.587 + rgb.b * 0.114; 
yuv.u = rgb.r * -0.169 + rg>.g * -0.331 + rgb.b * 0.5 + 128; 
yuv.v = rgb.r * 0.5 + rgb.g * -0.419 + rgb.b * -0.081 + 128; 



void YUVtoRGB (YUV &yuv, RGB &rgb) 
{ 

int y = yuv.y; 

int u = yuv.u - 128; 

int v = yuv.v - 128; 

rgb.r = y * 1.0 + u • 0.0 + v ♦ L402; 
rgb.g = y * 1.0 + u * -0.344 + v * -0.714; 
rgb.b = y * L0 + u * L772 + v • 0.0 ; 



void ClipRGB (RGB &rgb, unsigned max) 
{ 

if ((int) rgb.r < 0) 
rgb.r = 0; 

if (rgb.r > max) 
rgb.r = max; 

if ((int) rgb.g < 0) 
rgb.g = 0; 

if (rgb.g > max) 
rgb.g = max; 

if ((int) rgb.b < 0) 
rgb.b = 0; 

if (rgb.b > max) 
rgb.b = max; 

} 

unsigned long PackYUV (YUV &yuv) 
{ 

return (unsigned long) 

((((yuv.y >> YSHKTl) << YSHIFT2) & YMASK) + 
(((yuv.u >> USHIFT1) << USHIFT2) & UMASK) + 
(((yuv.v » VSHIFT1) << VSHIFT2) & VMASK)); 



void UnPackYUV (unsigned long PackedYUV, YUV &yuv) 
{ 

yuv.y = ((PackedYUV & YMASK) » YSHIFT2) << YSHDFT1; 
yuv.u = ((PackedYUV & UMASK) » USHIPT2) << USHIFT1; 
yuv.v = ((PackedYUV & VMASK) >> VSHEFT2) << VSHIFT1; 
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unsigned long YUVDifTerence (YUV &yuvl, YUV yuv2) 
{ 

unsigned difference; 

difference = abs (yuvl.y - yuv2.y); 
difference + = abs (yuvl.u - yuv2.u); 
difference + = abs (yuvLv - yuv2.v); 

return difference; 

} 

ASSERT JH: 
#ifiidef ASSERT_H 
#defme ASSERTH 

#undef assert 

#ifdefNDEBUG 

#define assert( ignore) ((void)O) 

#else 

#define assert(expr) ((expr)?(void)0:assei1^0,#expr > _FIlJ:_^_LINE )) 
#endif ~ 

extern void assert2 (int value, char *expr > char *file, unsigned lineno); 
#endif 

CURSOR.H; 
#i£hdef CURSORH 
#define CURSOR Ji 

class Cursor 
{ 

public: 

int X 0 [ return _x; } 
int Y () { return _y t } 

void SetMaxX (int maxx) { jnaxx = maxx; _x = _max_x / 2; } 

void SetMaxY (int max jr) { _max_y = maxjr, _y = jnax_y / 2; } 

void Set Width (unsigned width) { _width = width; } 

void SetColor (unsigned long color 1, unsigned long color2) 

{ _colorl = color 1; _color2 = color2; } 

void SetOffoetX (int offeet_x) { _offeet_x = of!set_x; } 

void SetOffsetY (int offeet_y) { _offeetjr = ofifeet_v; } 

void UpdateX (int x); 

void UpdateY (int y); 

void Draw (void (*LineDraw) (unsigned xl, unsigned yl, unsigned x2, unsigned y2, unsigned long color)); 

private:. 

urtjc; 

intjr, 

int_max_x; 

int_max_y t 

int_offset_x; 

int^offeetjr; 
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unsigned_width; 
unsigned longjsolorl; 
unsigned long_color2; 



#endif 

FOVEATOILH: 
#ifndef FOVEATORH 
#define FOVEATOR H 

#include "superpix.h" 

// This class will create an array of superpixels 

class Foveator 

{. 

public: 

Foveator (); 
-Foveator (); 

void SetScreenResolution (unsigned xjresolution, unsigned y_resolution); 

void SetHalfResoiution (float half_resolution) 

{ Jialfresolution = halfresolution; } 

void SetDistance (unsigned distance) 

{ .distance = distance; } 

void SetPixelsPerCM (unsigned pixels_per_cm) 

{ _pixels_per_cm = pixela_per_cm; } 

void SetWO (unsigned wO) 

{ _w0 s wO; } 

SuperPixel *Foveate (unsigned centers, 
unsigned centerjr, 
unsigned &super_pixel_count); 

private: 

void AddSuperPixels (unsigned centers, unsigned center_y, 

unsigned x, unsigned y, 

unsigned width, unsigned height); 
void AddSuperPixel (int x, int y, unsigned width, unsigned height); 
float half resolution; 
unsigned distance; 
unsigned _j>ixels_per_cm; 
unsigned _w0; 
unsigned _x_resolution; 
unsigned _y_resolution; 
SuperPixel *_euperjMxels; 
unsigned j3uper_pixel_index; 
unsigned _max_super_pixels; 

#endif 

INTEGRAT.H: 
#ifiidef INTEGRATED 
#defme INTEGRATE H 
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#include "palette.h" 
#include "superpixJi" 

extern void Integrate (SuperPixel *superjjixels, 
unsigned 8uper_pixel_count, 
void *source_nnage, 
unsigned image_width, 
Palette * palette, 
unsigned tgaJmageJype_code); 

#endif 

INTERPOL.H: 

#ifndef INTERPOLATE^ 

#de£ine INTERPOLATED 

#include "superpix.h" 

extern void Interpolate (SuperPixel *super_pixels, 
unsigned super j?ixel_count, 
char *dest_image, 
unsigned image_width, 
unsigned tga_image_type_code); 

#endif 

LEVEL.H: 
#i&def LEVELH 
#define LEVEL_H 

// A Level struct represents a level in a Laplacian Pyramid 

struct Level 
{ 

unsigned width; 
unsigned height; 
char ♦image; 

}; 

// A Threshold struct confines the contrast threshold region 
// for a level The region is not inclusive of x2,y2. 

struct Threshold 
{ 

unsigned xl; 
unsigned yl; 
unsigned x2; 
unsigned y2; 

}; 

//Reduce VtoV 

extern void Reduce2x2 (Level *a, Level *b); 
// Expand V to V 



70 



SUBSTITUTE SHEET (RULE 26) 



WO 98/33315 



PCT/US97/23737 



extern void Expand 3*2 (Level *a, Level *b, Threshold •threshold = 0); 
// Reduce 'a' to V 

extern void Reduce 3x3 (Level *a, Level *b); 
//Expand V to V 

extern void Expand3x3 (Level *a, Level *b, Threshold * threshold = 0); 
// b = a - b 

extern void Subtract (Level *a, Level *b, Threshold ♦threshold = 0); 
//b = a + b 

extern void Add (Level *a, Level *b, Threshold * threshold = 0); 
//b = a 

extern void Copy (Level *a, Level *b); 

#endif 
MOUSE.H: 
#ifhdef MOUSE H 
#define MOUSEJi 

#define MOUSE J-EOTJBUTTONMASK 0x01 
#define MOUSE_RIGHT_BUTTON_MASK 0x02 
#define MOUSE_MIDDIJ2_BUTTON_MASK 0x04 

extern int Mouselnit (); 

extern unsigned short MouseButtonStatus 0; 

extern unsigned long MousePosition (); // 3 1-16= column, 15-0= row 

extern unsigned long MouseMotion (); // 31-16 =xdir, 15~0=ydir 

#endif 
PALETTE:H: 
#ifhdef PALETTEH 
#define PALETTE_H 

^include "rgb.h" 

class Palette 
{ 

public- 
Palette (unsigned size = 256, unsigned bits_perjrimary = 8); 
-Palette (); 

void Set (unsigned index, unsigned greyjBcale_value); 

void Set (unsigned index, unsigned r, unsigned g, unsigned b); 

void Set (unsigned index, RGB &rgb); 

RGB &Get (unsigned index); 

unsigned Size () { returnjdze; } 
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unsigned BitaPerPrimary () { return _bits_per_j)rimary; } 

protected: 

RGB •_rgb_table; 

private: 

void LoadGreyScales 0; 

unsigned _aize; 

unsigned _bits_per_primary; 

>; 

#endif 
PROMPTS: 
#ifodef PROMPT_H 
#define PROMPTH 

void GetString (char * string); 
void GetFloat (float &result); 
void GetUnsigned (unsigned &result); 

#endif 
PTIMER.H: 
#ifhdef PTCMER_H 
#define PTTMEbTh 

// For some reason, the Watcom compiler generates a W627 warning 
// whenever this header is included. The following pragma will 
// turn off this warning. 

#pragma warning 627 9 

class Pentium Timer 
{ 

public: 

Pentium Timer (); 
void Start (); 
void Stop (); 

void Elapsed (unsigned long &high, unsigned long &low); 
unsigned long HighReg (); 
unsigned long LowReg (); 

private: 

int startedjlag; 

}; 

#endif 
PYRAMID.H: 
#imdef PYRAMID H 
#define PYRAMID" H 

#include "level.h" 

class Pyramid 
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{ 

public: 

Pyramid (unsigned base_width, 

unsigned base_height, 

unsigned levels, 

float contrast_threshold, 

float halfresolution, 

float dominantfreqO, 

float imagedistance, 

float pixels_perjcm); 
-Pyramid 0; 

void ComputeThresholds (unsigned x, unsigned y); 

protected: 

unsigned MaxEcc (unsigned level); 

unsigned Jevels; 
unsigned _base_width; 
unsigned Jiasejieight; 
Level *Jbig_levels; 
Level *__small_levels; 
Threshold ^thresholds; 
float _contrast_threshold; 
float _halfj:esoiution; 
float _domiiiant_freqO; 
float _image_distance; 
float __pixels_per_ cm; 

}; 

class EncodePyramid : public Pyramid 
{ 

public: 

EncodePyramid (unsigned base width, 

unsigned baseheight, 

unsigned levels, 

float contrastthreshold, 

float halfresolution, 

float dominant freqO, 

float image_distance, 

float pixels_per_cm); 
-EncodePyramid (); 
void AllocateLevels (Level *basejevel); 
void Reduce (unsigned level); 
void Expand (unsigned level); 
void Subtract (unsigned level); 
Level *GetLevel (unsigned level); 
Level ♦GetLevelN 0; 

h 

class DecodePyramid : public Pyramid 
{ 

public: 

DecodePyramid (unsigned base_width, 
unsigned base Jieight, 
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unsigned levels, 

float contrast_threshold, 

float halfresolution, 

float dominant_freqO, 

float image_distance, 

float pixels_per_cm); 
-DecodePyramid (); 
void AlIocateLevels (); 
void Expand (unsigned level); 
void Add (Level * diff erenced_le vel, unsigned level); 
Level *GetLevel (unsigned level); 
void SetLevelN (Level 'level); 

>; 

#endif 
RGB.H: 

#imdef RGB_H 
#define RGB_H 

struct RGB 
{ 

unsigned r; 
unsigned g; 
unsigned b; 

h 

#endif 

SUPERPDLH: 
#ifiidef SUPEKPIX_H 
#de£ine SUPERPDC_H 

struct SuperPixel 
{ 

//If the origin is at the bottom left of your view, then 
// X and Y are the bottom left of the super pixel. 

unsigned x; 
unsigned y; 
unsigned width; 
unsigned height; 
unsigned long color; 

>; 

#endif 
TGA.H: 

#imdef TGA_H 
#define TGA_H 

#include M palette.h n 

// Image descriptor byte masks 

#define SCREEN_ORIGIN_MASK 0x20 
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#define INTERLEAVE MASK OxCO 



*• 
#• 

V 



FILE STRUCTURE FOR THE ORIGINAL TRUEVISION TGA FILE 
FIELD 1 : NUMBER OF CHARACTERS IN ID FIELD (1 BYTES) 
FIELD 2 : COLOR MAP TYPE (1 BYTES) 
FIELD 3 : IMAGE TYPE CODE (1 BYTES) 
= 0 NO IMAGE DATA INCLUDED 
= 1 UNCOMPRESSED, COLOR-MAPPED IMAGE 
= 2 UNCOMPRESSED, TRUE-COLOR IMAGE 
= 3 UNCOMPRESSED, BLACK AND WHITE IMAGE 
= 9 RUN-LENGTH ENCODED COLOR-MAPPED IMAGE 
= 10 RUN-LENGTH ENCODED TRUE-COLOR IMAGE 
** = 11 RUN-LENGTH ENCODED BLACK AND WHITE IMAGE 

*• FIELD 4 : COLOR MAP SPECIFICATION (5 BYTES) 
** 4.1: COLOR MAP ORIGIN (2 BYTES) 

** 4.2: COLOR MAP LENGTH (2 BYTES) 

4.3: COLOR MAP ENTRY SIZE (2 BYTES) 
FIELD 5 : IMAGE SPECIFICATION (10 BYTES) 
** 5.1 : X-ORIGIN OF IMAGE (2 BYTES) 

** 5.2: Y-ORIGIN OF IMAGE (2 BYTES) 

** 5.3 : WIDTH OF IMAGE (2 BYTES) 

** 6.4 : HEIGHT OF IMAGE (2 BYTES) 

** 5.5: IMAGE PIXEL SIZE (1 BYTE) 

** 5.6: IMAGE DESCRIPTOR BYTE (1 BYTE) 

** BITS 0 -3 : ATTRIBUTE BITS PER PIXEL 

** BIT 4 : LEFT TO RIGHT PIXEL ORDERING 

** BIT 5 : TOP TO BOTTOM PIXEL ORDERING 

BIT 6 : 0 
** BIT 7 : 0 

*♦ FIELD 6 : IMAGE ID FIELD (LENGTH SPECIFIED BY FIELD 1) 

** FIELD 7 : COLOR MAP DATA (BIT WIDTH SPECIFIED BY FIELD 4.3 AND 

** NUMBER OF COLOR MAP ENTRIES SPECIFIED IN FIELD 4.2) 

** FIELD 8 : IMAGE DATA FIELD (WIDTH AND HEIGHT SPECIFIED IN 

•* FIELD 5.3 AND 5.4) 



ALL BYTE ORDERING IS LITTLE-ENDIAN 



struct TGAHeader 
< 

unsigned char idjieldjength; 
unsigned char color map type; 
unsigned char image type code; 
unsigned short color_map_origin; 
unsigned short color_map_length; 
unsigned char color_map_entry_aize; 
unsigned short x_origin; 
unsigned short y_origin; 
unsigned short width; 
unsigned short height; 
unsigned char bits_per_pixel; 
unsigned char image descriptor; 

}; 
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enum TGAImageTypeCode 
{ 

TGA_COLOR_MAPPED = 1, 
TGA_TRUE_COLOR = 2, 
TGA GREY SCALE = 3 

}; 

class TGA 
{ 

public: 
TGA (); 
-TGA (); 

int Load (char *szFilename); 

unsigned Width 0 { return J;ga_header. width; } 

unsigned Height () { return_tga_header. height; } 

void * Image () { return_image; } 

Palette *GetPalette () { return_palette; } 

unsigned ImageTypeCode () { return _tgaJieader.image_type_code; } 
unsigned BitsPerPixel () { return _tga_header.bits_perj>ixel; } 
unsigned BytesPerPixel () { return (_tga_header.bits_per_pixel + 7) / 8; } 

private: 

char *_image; 
unsigned width; 
unsigned ^height; 
TGAHeader _tga_header; 
Palette *_palette; 

>; 

#endif 
VESAH: 
#undef VESA_H 
#define VESA^H 

#include "palette Ji" 
#inciude "svga.h" 
#include "vesavbe.h" 
#include -vbeafh" 

typedef SV_devCtx VESADeviceContext; 

extern int VESAInit (unsigned x, unsigned y, unsigned depthjrits, int linear Jmffer = 1); 
extern void VESAClose (); 

extern const VESADeviceContext *VESAGetDeviceContext (); 
// All Scitech palettes should have 8 bits per primary 
extern void VESALoad (Palette &palette); 

extern void VESALine (unsigned xl f unsigned yl, unsigned x2, unsigned y2, unsigned long color); 
extern void VESABeginLine (); 
extern void VESAEndLine (); 

extern void VESALineFast (unsigned xl, unsigned yl, unsigned x2, unsigned y2, unsigned long color); 
extern void VESAPoint (unsigned x, unsigned y, unsigned long color); 
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extern void VESABeginPoint (); 
extern void VESAEndPoint (); 

extern void VESAPointF ast (unsigned x, unsigned y, unsigned long color); 
extern void VESAText (char *azText, unsigned x, unsigned y, unsigned long color); 
extern void VESAScreenBlt (char *bitmap, unsigned long length); 
extern void VESABitBlt (void *bitmap, unsigned width, unsigned height, 

unsigned bytes_j>er jrixel, unsigned x_ofiset, unsigned y_oflset); 
extern void VESAClear (unsigned long color); 

#endif 
YUV.H: 

#imdefYUVH 
#defineYUVJH 

I* 

Convert R'G'B' (tristimulus values) to VUV (color difference 
components). 

YTJV is used loosely to mean any color difference coding, 
whether it's VCBCR, YTBPR, or whatever else. 

Here, the conversion matrix for YTBPR will be used. 

To convert from tristimulus values to color difference components: 

[ Y* 601 J [ 0.299 0.587 0.114 1 [ R' ] 
[ B'-V 601 ]=[-0.299 -0.587 0.886 ]*[ G* ] 
[ R'-Y* 601 J [ -0.701 -0.587 -0.114 1 [ B' ] 

Scale PB and PR ranges to [-0.5..+0.5] to get: 

[ T 601 1 [ 0.299 0.587 0.114 ) [ R' ) 

[ PB 601 1= [-0.168736 -0.331264 0.5 ]*[ G* 1 

I PR 601 ] [ 0.5 -0.418688 -0.081312 ][B f ] 

The first row, the luma coefficients, sum to L The second and 
third rows, the chroma coefficients, each sum to zero. 

Here is the inverse matrix: 

[ R' 1 [ 1.0 0.0 L402 ] [ r 601 3 

I G* ]=[ 1.0 -0.344136 -0.714136 ]*[ PB 601 ] 

[ B* ] [ L0 L772 0.0 ] [ PR 601 ] 

•/ 

#include "rgb.h" ■ ~ - 

#include "palette.h" 

#define YUV1055 
//#define YUV844 
//#define YUV633 
//#define YUV422 
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#if defined (YUV1055) 
// 10:5:5 

#define YWIDTH 10 
#define UWIDTH 5 
#define VWIDTH 5 
#define YMASK QxOOOFFCOO 
#define UMASK OxOOOOOSEO 
#define VMASK 0x000000 IF 
#de£ine YSHIFT1 0 
#define USHIFT1 0 
#define VSHEFT1 3 
#define YSHIFT2 12 
#define USHIFT2 2 
#define VSHIFT2 0 

#elif defined (YUV844) 

// 8:4:4 

#define YWIDTH 8 
#define UWIDTH 4 
#define VWIDTH 4 
#define YMASK OxFFOO 
#define UMASK OxOOFO 
#defineVMASK OxOOOF 
#define YSHIFT1 0 
#define USHIFT1 0 
#define VSHIFT1 4 
#define YSHEFT2 8 
#define USHIFT2 0 
#define VSHIFT2 0 

#elif defined (YUV633) 

// 6:3:3 

#define YWIDTH 6 
#define UWIDTH 3 
#define VWIDTH 3 
#define YMASK OxFCO 
tfdefine UMASK 0x038 
#defineVMASK 0x007 
#define YSHIFT1 0 
#define USHIFT1 2 
#define VSHIFT1 5 
#define YSHEFT2 4 
#define USHIFT2 0 
#define VSHIFT2 0 

#elae 
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// 4:2:2 

#define YWTDTH 4 
#define UWIDTH 2 
#define VWIDTH 2 
#define YMASK OxFO 
#define UMASK OxOC 
#defineVMASK 0x03 
#define YSHIFT1 0 
#define USHUT1 4 
#define VS HIFT1 6 
#define YSHIFT2 0 
#define USH3FT2 0 
#define VSHIFT2 0 

#endif 

struct YUV 
{ 

unsigned y; 
unsigned u ; 
unsigned v; 

>; 

extern void YUVMapInit (Palette &palette); 

extern unsigned YUVFindClosestRGB (RGB &rgb, Palette &palette); 

extern void RGBtoYUV (RGB &rgb f YUV &yuv); 

extern void YUVtoRGB (YUV &yuv, RGB &rgb); 

extern void ClipRGB (RGB &rgb, unsigned max); 

extern unsigned long PackYUV (YUV &yuv); 

extern void UnPackYUV (unsigned long packed_yuv, YUV &yuv); 

extern unsigned long YUVDifTerence (YUV &yuvl, YUV yuv2); 

#endif 
ARGS.CXX: 
#include <math.h> 
#include <stdio.h> 
#include <stdlib.h> 
#include <string.h> 

#include "argsJi" 
#include "prompt.h" 

const char *usage_text = 

"usage: pyramid [proffleJrames]\n\n" 

" profile frames Fovea te 'profile frames 1 number of frames\n" 
.«■ "\n" 

"Click the left mouse button to toggle foveation.\n" 
"\n" 

"Press any key to quit the program. \rT, 

Arga::Arga (int argc, char **argv) 
{ 

^filename = new char [256]; 
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if (!_filename) 

throw "Args::ArgsO: memory allocation error"; 

strcpy (_filename, "default.tga"); 

_profile_frames = 0; 
Jialfresolution = 2.5; 
_image_distance = 12 * 2.54; 
_screen_width = 12 * 2.54; 

switch (argc) 
{ 

case 2: 

if (argv[ll[03 == T| argvdirO) == V| argv[l][0] == T) 
throw usage_text; 

_profile_frames = atoi (argv[l]); 

if (!_profile_frames) 
throw uaage_text; 

// Fall through to next case 

case 1; 
break; 

default: 

throw usage_text; 

> 

// Prompt the user for parameters 

printf ("\nFilename f%s]:\t", Jnename); 
GetString (^filename); 

printf ("Half resolution [%0.1f]:\t", _half_resolution); 
GetFloat (Jialf_resolution); 

printf ("Screen distance [%d]:\t M , _image_distance); 
GetUnsigned (_image_distance); 

printf ("Screen width [%dj:\t" t jscreenjwidth); 
GetUnsigned (__screen_width); 

//Add a .TGA to the file if it needs it 

if ((strlen <v filename) < 5) | 

(strcmpi ( & filename [strlen (filename) - 4], ".TGA"))) 
strcat (_filename, ".TGA"); 

FILE *fp = fopen (^filename, V); 
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ifdfp) 
{ 

// If the file is not in the current directory, and 

// the environment variable "IMAGEDIR" exists, then 

// prepend it to the filename. 

char *ev = getenv (TMAGE^Dm"); 

if (ev) 
{ 

char *temp = new char [256]; 
if (Itemp) 

throw n Args::Args(): memory allocation error 0 ; 

strcpy (temp, ^filename); 
strcpy (^filename, ev); 

// Add a pathname separator if needed 

if (__filename[strlen (_filename) - 1] != *\\'&& 
_filename{strlen (^filename) - 1] != '/*) 
strcat (_filename, "\\"); 

strcat ( filename, temp); 

fp = fopen ( filename, V); 

// If you still can't find the file, copy original name back 
ifdfp) 

strcpy (^filename, temp); 
else 

fclose (fp); 

delete [] temp; 

} 

} 

else 
{ 

fclose (fp); 

} 

} 

ArgB::-ArgB () 
{ 

delete 0 ^filename; 

} 

void ArgsnDump (unsigned width_pixels) 
{ 

printf ("%18s %0.1f degree%e\n\ 
"Half resolution:", 
_halfjresolution, 
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__half resolution == 1 ? " " : V); 
printf ("%18s %d cm\n", 

"Image distance: 0 , 

_image_distance); 
printf ("%18s %.2f pixels/cm\n ,, f 

"Image resolution:", 

((float) width_pixels) / jBcreen_width); 
printf ("%18s %0.2f (%0.2f degrees)\n", 

"Foveal pixel arc:", 

atan ((((float) jjcreen_width) / width_pixels) / _image_distance) * (360 / (2 *asin (L0))) * 60, 
atan ((((float) _screen_width) / width_pixels) / jmage^distance) • (360 / (2 *asin (L0)))); 

TEST.CXX: 
#include <conio h> 
#include <malloc.h> 
#include <mathii> 
#include <stdio.h> 
#indude <stdlih.h> 
#include <string.h> 
#include <time.h> 

#include "argB.h B 
#include "cursor.h" 
#include "mouse.h" 
#include "palette.h a 
#include n p timer .h" 
#include "pyramid.h" 
#include M tga.h" 
#include "vesa.h tt 

void aasert2 (int value, char *expr, char *file, unsigned lineno) 

if (value) 
return; 

VESAClose 0; 

fprintf (stderr, *%e:%d (%s)\n", file, lineno, expr); 
throw "Assertion failed"; 

> 

int main (int argc, char **argv) 

{ 

try 
{ 

ArgB arga (argc, argv); 
//Mouse 

printf ("Initializing mouse...\n"); 

if (Mouselnit ()) 

throw "Mouse driver not installed"; 
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int left_mouse_button = 0; 
//Image 

printf ("Loading %s...\n a t args.Filename ()); 
TGAtga; 

if (tgaJLoad (args filename ())) 

throw "Could not open specified TGA file"; 

unsigned image_width = tga. Width (); 
unsigned image_height = tga.Height (); 

// Graphics 

const unsigned S CREEN WTDTH = 1024; 
const unsigned SCREENHEIGHT = 768; 

if (image_width > SCREEN_WIDTH | image Jieight > SCREEN_HEIGHT) 
throw The image is too large to be displayed at this resolution"; 

if (VESAInit (SCREENWIDTH, SCREENHEIGHT, tga.Bit&PerPixel ())) 
throw "Can't initialize VESA driver"; 

//Cursor 

Cursor cursor; 

cursor.SetWidth (26); 
cursor.SetColor (255, 0); 
cursor.SetMaxX (image_width); 
cursor.SetMaxY (image height); 

cursor.SetOfisetX ((SCREEN WIDTH - image_width) / 2); 
cursor.SetOfifletY ((SCREEN_HEIGHT - image_height) / 2); 

//Palette 

switch (tgaJmageTypeCode ()) 
{ 

case TGA_COLOR_MAPPED: 

case TGA_TRUE_COLOR: 

throw "Only greyscale images are supported"; 

case TGA_GREY_SC ALE : 
{ 

Palette palette; // Default greyscale palette 
VESALoad (palette); 

} 

break, 
default: 
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throw "Invalid TGA type code"; 

} 

VESABitBlt (tgaJmage (), 
imagejvidth, 
image_height, 
tga.BytesPerPixel (), 
SCREEN_WIDTH / 2 - image_width / 2, 
SCREEN~HEIGHT / 2 - imageheignt / 2); 

// Misc. flags and counters 

int foveationjlag = 1; 
unsigned long totalframes = O, 
unsigned long totalticks = 0; 
PentiumTimer pentium_timer; 
unsigned total_kilocyclefl = 0; 
unsigned long time _hi, timer Jo; 

srand (0); 

//Pyramids 

const unsigned LEVELS = 6; 
int heapchk = _heapchk (); 

if (heapchk != _HEAPOK && heapchk != _HEAPEMPTY) 
throw "Heap error"; 

const float CONTRASTJTHRESHOLD = 0.01; 
const float DOMINANTFREQUENCYO = 24; 

EncodePyramid EP (imagB_width, 
imageheight, 
LEVELS, 

CONTRASTJTHRESHOLD, 
args.HalfResolution (), 
DOMINANTFREQUENCYO, 
args.Distance (), 

SCREEN jmUTH I args-Width ()); 
DecodePyramid DP (image_width, 
imageheight, 
LEVELS, 

CONTRASTTHRESHOLD, 
args.HalfResolution 0, 
DOMINANTFREQUENCYO, 
argsDistance (), 

SCREENJWIDTH / arga.Width ()); 
Level levelO; 

level_0. image = (char *) tgaJmage (); 
levelj). width - image_width; 
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levelO.height = image_height; 

EP JUlocateLevels (&level_0); 
DP^AUocateLevels (); 

heapchk = _heapchk (); 

if (heapchk != JKEAPOK && heapchk != _HEAPEMPTY) 
throw "Heap error 0 ; 

for (unsigned level = 0; level < LEVELS - 1; level ++) 
EP.Reduce (level); 

// Swallow any accidental keystrokes 

while (kbhit 0) 
getchO; 

// Main loop 

for (;;) 
{ 

long mouse_motion; 

if (args.ProfileFrames ()) 
{ 

if (total frames == args.ProfileFrames ()) 
break; 

moueemotion = (rand () % image_width) - image_width / 2; 
mousemotion <<= 16; 

mouse motion += (rand () % image_height) - imagejieight / 2; 

else 
{ 

unsigned short mousejstatus = MouseButtonStatus (); 

if (lleftmouse button && (mouse_status & MOUSE LEFT BUTTON MASK)) 
{ ~ " 

leftjnousejbutton = 1; 

foveationjlag = !foveation_flag; 

} 

if (!(mouse_status & MOUSEJJ5FT_BUTTON_MASK)) 
leftjnousejbutton = 0; 

mouse motion = MouseMotion 0; 

} 

if (kbhit ()) 
{ 

// swallow character 
getchO; 
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break; 



if (mouse_motion | left_mouse_button) 
{ 

// Change has occurred 

cursor.UpdateX ((short) (mouse_motion > > 16)); 
cursor. UpdateY ((short) (mouse_motion)); 

if ( fovea tion_flag) 
{ 

clock_t start_ticks = clock (); 
pentiumtrmer. Start (); 

// Foveate the image 

EP.ComputeThresholds (cursor X (), cursor.Y 0); 
DP.ComputeThreaholds (cursor X (), cursor.Y 0); 

//Encode 

for Oevel = LEVELS - 1; level > 0; level-) 
EP.Expand (level); 

for Oevel = 0; level < LEVELS - 1; level++) 
EP.Subtract Oevel); 

//Decode 

DP.SetLevelN (EP.GetLevelN 0); 

for Oevel = LEVELS - 1; level > 0; level-) 
{ 

DP.Expand Oevel); 

DP-Add (EP.GetLevel Oevel - 1), level - 1); 

> 

pentiumJdmer.Stop (); 
pentium_timer.Elapsed (timer_hi, timer Jo); 

totaljicks += clock () - start Jacks; 
+ +total_frames; 

if (timer_hi) 

throw "Pentium timer overflow*; 

total Jrilocycles + = time Jo / 1000; 

VESABitBlt (DP.GetLevel (0)->image, 
image_width, 
image Jiaight, 
tgaJByteePerPixel (), 
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(SCREEN_WIDTH >> lMimage_width >> 1), 
(SCEEENHEIGHT >> lHimage_height >> !))• 

} 

else 
{ 

VESABitBlt (tgaJmage (), 
image_width, 
image^height, 
tga.BytesPerPixel (), 
S CREENWIDTH / 2 - image_width / 2, 
SCREEN HEIGHT / 2 - image Jieight / 2); 

> 

cursor Draw (VESALine); 

} 

} 

VESAClose (); 

printf ("%18e 9te\n", "Filename:", arga.Filename ()); 

printf ("%18e %d X %d", ■Dimensions:", image_width, image Jieight); 

switch (tga . TmageTypeCode ()) 
{ 

case TGA_COLOR_MAPPED: 
printf ("color mapped image\n n ); 
break; 

case TGA TRUE COLOR: 
printf ("true color image\n"); 
break; 

case TGAGREYJSCALE: 
printf ("grayscale image\n"); 
break; 

default: 

printf ("(unknown image type code)\n°); 

} 

args.Dump (SCREEN JWIDTH); 

if (total frames != 0) 
{ 

printf ("%18s %d\n", 

"Total frames:", 

totalframes); 
printf (*%18e %.2f seconds\n", 

"Total time:", 

((float) totolticks) / CLOCKS PER SEC); 
printf ("%18s %0.2f frames/Bec\n"7 

n m 

9 

((float) totalframes) / (((float) total_ticks) / CLOCKS_PER_SEC)); 
printf ("%18s %0.0f pixels/sec\n", 
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((float) totaljrames * image_width * image Jieight / (((float) totaljicks) / CLOCKS_PER_SEC)); 
printf ("%18s %0.0f megacycles\n°, 

"Pentium Cycles:", 

((float) totaljrilocycles) / 1000); 
printf ("%18s %0.0f megacydea/frame\n" t 

it m 

i 

(((float) total_kilocycles) / 1000) / totaljrames); 
printf ("%18s %0.1f cydes/pixelVAn", 

« H 

I 

((float) totaljrilocycles) / totaljrames / image_width / image Jieight * 1000); 



return 0; 

} 

catch (const char *msg) 
{ 

VESAClose (), 

fprintf (stderr, "fteVAn", msg); 
return -1; 

> 

> 

AR6S.H: 
#ifhdef ARGSH 
#define ARGS_H 

class Args 
{ 

public: 

Args (int argc, char **argv); 
-ArgsO; 

char 'Filename 0 { return Jilename; } 
float HalfResolution () { return half resolution; } 
unsigned Distance () { return Jmage^distance; } 
unsigned Width () < return jxareenwidth; } 
unsigned ProfileFrames () { returrTproffle^frames; } 
void Dump (unsigned widthjrixels); 

private: 

char * Jilename; 
float Jialfresolution; 
unsigned Jmage distance; 
unsigned _screen_width; 
unsigned _profile frames; 

>; 

#endif 
ARGS.CXX: 
#include <math.h> 
#include <stdio.h> 
#include <stdlib.h> 
#include <string.h> 
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#include B args.h n 
#include "prompt.h" 

const char *usage_text = 

"usage: superpix [profile_frames] \n\n° 

" profile frames Fovea te 'profile_frames' number of frames\n" 

V 

"Click the left mouse button to toggle foveation.\n" 
V 

"Press any key to quit the program. \n"; 

ArgB::ArgB (int argc, char **argv) 
{ 

_filename = new char [256]; 

if (!_filename) 

throw "ArgB:^Args(): memory allocation error"; 

strcpy (^filename, "defaulttga"); 

_profile__frames = 0; 

halfresolution = 2.5; 
_image_distance = 12 * 2.54; 
_w0 = 1; 

jBcreen_width = 12 * 2.54; 

switch (argc) 
{ 

case 2: 

if (argvUHOJ == 7 I argv[lj[0] == v | argv[l][0] == T) 
throw usage_text 

_profile_frames = atoi (argv[l]); 

if (profile Jrames) 
throw usage_text; 

// Fall through to next case 

case 1: 
break; 

default: 

throw usagetext; 

} 

// Prompt the user for parameters • * 

printf ("\nFilename [%n):\t n , ^filename); 
GetString (^filename); 

printf ("Half resolution [%0.1f]:\t", Jhalf_resolution); 
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GetFloat (_half_resolution); 

printf ("Fovea! pixel size [%d]:\t n , _w0); 
GetUnsigned (_w0); 

printf ("Screen distance [%d]:\t", _image_distance); 
GetUnsigned (_image_distance); 

printf ("Screen width f%d]:\t\ _screen_width); 
GetUnsigned (_screen_width); ~ 

//Add a TGA to the file if it needs it 

if ((strlen (^filename) < 5) | 

(strcmpi (&Jilename[strlen (_filename) - 4], ".TGA"))) 
strcat (^filename, ".TGA"); 

FILE *fp = fopen (^filename, V); 

if(!fp) 
{ 

// If the file is not in the current directory, and 

// the environment variable TMAGEDER" exists, then 

// prepend it to the filename. 

char *ev = getenv ("IMAGE JDIR"); 

if (ev) 
{ 

char *temp = new char [256]; 
if (Itemp) 

throw n Args:LArgB(): memory allocation error"; 

strcpy (temp, ^filename); 
strcpy (^filename, ev); 

// Add a pathname separator if needed 

if (_filenamefstrlen ^filename) - 1] != '\\' && 
Jilenamefstrlen (jBlename) - 1] != 7) 
strcat (^filename, "\\"); 

strcat (^filename, temp); 

£p = fopen (Jilename, V); 

// If you still can't find the file, copy original name back 

ifdfp) 

strcpy (Jilename, temp); 

else 

fclose (fp); 
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delete □ temp; 

} 

> 

else 
{ 

fclose (fp); 

} 

} 

ArgB::-Args () 
< 

delete D filename; 

) 

void Args::Dump (unsigned width_pixels) 
{ 

printf ("%18s %0.1f degrees5to\n B , 

"Half resolution:", 

_halfresolution, 

Ihalfjresolution == 1 ? " ■ : V); 
printf ("%18s %d pixeWte\n", 

"Foveai pixel size:", 

_w0, 

~w0 1?"": V); 
printf ("%18s %d cm\n", 

"Image distance:", 

_image_distance); 
printf ("%18s %.2f pixela/cm\n", 

"Image resolution:", 

((float) width_pixels) / _screen_width); 
printf ("%18s %0.2f (%0.2f degrees)\n", 

"Foveai pixel arc: 0 , 

a tan ((((float) _screen_width) / width_pixels) / 
atan ((((float) screen_width) / width_pixels) / 

} 

TEST.CXX: 
#include < assert. h> 
#include <conio.h> 
#indude <math.h> 
#include <stdio.h> 
#include <stdlibii> 
#include < string Ji> 

#include "argsJi" 
#include "cursor-h" 
#include "foveator.b" 
#include "integrat.h" 

rs 9 #include "interpoLh" . . . v-- 

#include "mouse .h" 
#include "palette.h R 
#include "ptimer Ji" 
#include "tga-h" 
#indude "vesaJi" 
#include "yuviT 
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#include "ztimer Ji n 

int main (int argc, char **argv) 
{ 

try 

{ 

Args args (argc, argv); 
// Mouse 

printf ("Initializing mouse.. .\n"); 

if (Mouselnit ()) 

throw "Mouse driver not installed"; 

int leftjnouse_button = 0; 

//Image 

printf ("Loading %s...\n\ args.Filename ()); 
TGAtga; 

if (tgaXoad (args.Filename ())) 

throw "Could not open specified TGA file"; 

unsigned image_width = tga. Width (); 
unsigned image_height = tga- Height (); 

// Graphics 

const unsigned SCREEN_WIDTH = 1024; 
const unsigned SCREENJHEIGHT = 768; 

if (image_width > SCREEN_WIDTH | image Jieight > SCREEN_HEIGHT) 
throw "The image is too large to be displayed at this resolution"; 

if (VESAInit (SCREEN JWIDTH, SCREEN JIEIGHT, tga.BitsPerPixel ())) 
throw "Can't initialize VESA driver"; 

//Cursor 

Cursor cursor, 

cureor.SetWidth (26); 
cursor.SetColor (256, 0); 
cursor.SetMaxX (image_width); 
cursor.SetMaxY (image Jieight); 

cursor.SetOffsetX ((SCREEN_WIDTH - image_width) / 2); 
cursor.SetOffsetY ((SCREEN JIEIGHT - image Jieight) / 2); 

//Palette 

Palette *palette; 
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switch (tga.ImageTypeCode ()) 
{ 

case TGA_COLOR_MAPPED : 

palette = tga.GetPalette 0; 

VESALoad (*palette); 
YUVMapInit (*palette); 

break; 

case TGA_TRUE_COLOR: 

cursor.SetColor (QxOOFFFFFF, 0x00000000); 

break; 

case TGA GREYSCALE: 
{ 

Palette palette; // Default greyscale palette 
VESALoad (palette); 

} 

break; 
default: 

throw Invalid type code"; 

> 

VESABitBlt (tgaJmage 0, 
image_width, 
image_height, 
tga.ByteaPerPixel (), 
SCREEN_WIDTH / 2 - image_width / 2, 
SCREEN_HEIGHT / 2 - image Jieight / 2); 

// Foveator 

Fovea tor foveator, 

foveator. SetS creenResolution (image_width, image_height); 
foveator. SetHalfResolution (args.HalfResolution ()); 
foveator.SetDistance (args.Distance ()); 
foveator.SetPixelsPerCM (SCREEN_WIDTH / args. Width ()); 
foveator.SetWO (args.WO ()); 

unsigned superj>ixel_count; 
SuperPixel *super_pixels; 

char * foveated image = new char [image_width • image_height * tga.ByteaPer Pixel ()]; 

if (ifoveatedimage) 

throw "Memory allocation error"; 

// Misc. flags and counters 
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int foveationjlag = 1; 
unsigned long totalframes = 0; 
unsigned long total_ticks = 0; 
unsigned long total_super_pixels = 0; 

ZTimerlnit 0; 

srand (0); 

PentiumTimer pentium timer; 
unsigned totalkilocyclea = 0; 
unsigned long timerju, tinier Jo; 

// Swallow any accidental keystrokes 

while (kbhit ()) 
getchO; 

// Main loop 

for(;;) 

long mouse_motion; 

if (args.ProffleFrames 0) 
{ 

if (total frames == args.ProfileFrames ()) 
break; 

mouse_motion = (rand () % image_width) - image_width / 2; 
mouse_motion <<= 16; 

mouse_motion += (rand 0 % image height) - image height / 2; 

} 

else 
{ 

unsigned short mousejrtatus = MouseButtonStatus (); 

if (!left mouse button && (mouse status & MOUSE LEFT BUTTON MASK)) 
i 

left_mouse_button = 1; 

fovea tionflag = ! foveationjlag; 

} 

if (!(mouse__8tatus & MOUSE J-EFT_BUTTON_MASK)) 
left_mouse_button = 0; 

mouse motion = MouseMotion 0; 

} 

if (kbhit ()) 
{ 

// swallow character 
getch (); 
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break; 

} 

if (mouse motion | left_mouse_butt»n) 
{ 

// Change has occurred 

cursor. UpdateX ((ahort)(mouse_motion > > 16)); 
cursor.UpdateY ((shortXmouse_motion)); 

if (foveation flag) 
{ 

LZTimerOn (); 
pentiumjimer. Start (); 

superj>ixels = foveator.Foveate (cursor X (), 
cursor.Y (), 
super_pixel_count); 

total_Buper_pixelfl += super jrixel_count; 

Integrate (euper_pixels, 
super j>ixelj»unt, 
tgaJmage 0, 
image^width, 
palette, 

tgn , TmsgeTypeCode ()); 

Interpolate (super_pixels, 
super_pixel_count, 
foveatedjmage, 
image_width, 
tgaJmageTypeCode ()); 

LZTimerOffO; 
pentium_timer.Stop (); 
pentium_tiiner.Elapsed (timer_hi, timer Jo); 
total Jacks += LZTimerCount {); 
+ +total_frames; 

if (timer_hi) 

throw "Pentium timer overflow"; 

total Jrilocycles += timer Jo / 1000; 

VESABitBlt (foveatedjmage, 
image_width, 

image Jieight, -t »••"■•■" 

tga.ByteflPerPixel (), 

(SCREEN WIDTH >> 1) - (image_width » 1), 
(SCREENHEIGHT » 1) - (image Jieight >> 1)); 

} 

else 
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{ 

VESABitBlt (tga. Image (), 
image_width, 
image_height, 
tga-BytesPerPixei 0, 
SCREENWIDTH / 2 - image_width / 2, 
SCREEN HEIGHT / 2 - image^ height / 2); 

) 

cuirsorJDraw (VESALine); 

} 

} 

VESAClose (); 

printf ("%17s: %e\n", "Filename", args.Filename ()); 

printf ("%17s: %d X %d •Dimensions", imagejridth, imageheight); 

switch (tga TmageTypeCode ()) 
{ 

case TGA_COLOR_MAPPED : 
printf ("color mapped image\n"); 
break; 

case TGA_TRUE_COLOR: 
printf ("true color image\n"); 
break; 

case TGA_GREY_SC ALE : 
printf ("greyscale image \n"); 
break; 

default: 

printf ("(unknown image type code)\n"); 

} 

printf ("%17s: %0.1f degree%s\n", 

"Half resolution", 

argB JEIalfResolution (), 

argsiialfResolution () 1 ? " " : V); 
printf ("%17s: %d pixel%s\n", 

"Foveal pixel size", 

arga.W0 (), 

ar^-WO ()==!?"": V); 
printf ("%17s: %d cm\n", 

"Image distance", 

argsJDifltance ()); 
printf ("%17s: %.2f pixels/cm\n", * 

"Image resolution", 

((float) SCREEN JVIDTH) / args. Width ()); 
print ("%17s: %0.2f 1%0J2£ degrees)\n", 
"Fovea! pixel arc", 

atan ((((float) args. Width ()) / SCREENWIDTH) / args.Distance ()) * (360 / (2 * asin (L0))) * 60, 
atan ((((float) ar^Width ()) / SCREEN_WIDTH) / args-Distance 0) * (360 / (2 * asin (L0)))); 
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if (total frames != 0) 
{ 

printf ("%17s: %d\n", 

"Total frames", 

total_framefl); 
printf ("%17s: %.2f seconds\n", 

"Total time", 

(double) total_tick3 * LZTIMER RES) ; 
printf ("%17s %&2f rrames/Bec\n", 

n m 

> 

((float) totaljrames) / LZTIMER_RES / totaljacks); 
printf ("%17s %0.0f pixelsfeec\n", 

» n 

I 

((float) totaljrames * imagejwidth * image Jieight) / LZTCMER_RES / total_ticks); 
printf ("%17s: %d ep/frame\n", 
"Avg superpixels", 
total_euper_pixelfl / total_frames); 

printf C%11b: %d:l\n", 
"Compression", 

(image_width * image_height) / (total_euper_pixels / totaljrames)); 
printf ("%17s: %0.0f megacycles\n", 

"Pentium Cycles", 

((float) totaljrilocycles) / 1000); 
printf ("%17s %0.1f megacydes/rrame\n", 

m m 

» 

(((float) total Jrilocycles) / 1000) / total_£rames); 
printf ("%17b %0.1f cycles/pixel\n", 

n m 

i 

((float) totaljrilocycles) / totaljrames / image width / image height * 1000); 

> 

delete D foveatedimage; 
return 0; 

} 

catch (const char *msg) 
{ 

VESAClose (); 

fprintf (stderr, "%s\n n f mag); 
return -1; 

} 

} 

ARGS.H: 
#imdef ARGS H 
#define ARGS ~H 

class Arga 
{ 

public: 
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Args (int argc, char **argv); 
-Args 0; 

char ^Filename 0 { return _filename; } 

unsigned WO () { return _wO; } 

float HalfResolution 0 { return halfresolution; } 

unsigned Distance () { return _image_distance ; } 

unsigned Width () { return __screen_width; } 

unsigned ProfileFrames () { return __profile_frames; } 

void Dump (unsigned width_pixels); 

private: 

char * filename; 
unsigned _w0; 
float _half_resolution; 
unsigned image distance; 
unsigned jBcreen_width; 
unsigned profile frames; 

}; 

#endif 
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CLAIMS 

We claim: 

1. A system for conveyance of video image data from a first 
computer system to a second computer system, the system comprising: 

a first computer system for real-time foveated . 
transformational coding of video image data; 

a second computer system for decoding the foveated 
transformationally coded video image data; and 

a communications channel linking the first computer system 
and the second computer system. 

2. The system of claim 1 in which the real-time foveated 
transformational coding of video image data includes manipulation of a 
subset of the video image data selected by a first foveation function. 

3. The system of claim 2 in which the first foveation function 
corresponds to a first human perception response to first image data 
derived from the video image data. 

4. The system of claim 2 in which the first foveatioii function 
corresponds to a first human perception spatial frequency response to the 
first image data and a first human perception eccentricity visual response 
to the first image data. 

5. The system of claim 3 or 4 in which the first foveation 
function is a function of a current gaze point. 

6. The system of claim 1 in which the real-time foveated 
transformational coding of the video image data includes threshold 
filtering of second image data derived from the video image data. 
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7. The system of claim 6 in which the threshold filtering of the 
second image data is based upon a second foveation function. 

8. The system of claim 7 in which the second foveation function 
corresponds to a second human perception response to the second image 
data. 

9. The system of claim 7 in which the second foveation function 
corresponds to a second human perception spatial frequency response to 
the second image data and a second human perception eccentricity visual 
response function to the second image data. 

10. The system of claim 8 or 9 in which the second foveation 
function is based upon a current gaze point. 

11. A system for conveyance of video information from a first 
computer system to a second computer system, the system comprising: 

a first computer system for foveated transformational coding 
of video image data; 

a second computer system for decoding the foveated 
transformationally coded video image data; 

a communications channel linking the first computer system 
and the second computer system; and 

wherein the foveated transformational coding of the video 
image data consists of foveated Laplacian pyramidal 
transformational coding, foveated wavelet pyramidal 
transformational coding, or foveated block discrete cosine 
transformational coding. 

12. The system of claim 11 in which the foveated 
transformational coding of the video image data is performed in real-time. 
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13. The system of claim 11 in which the foveated tranformational 
coding of the video image data includes manipulation of a foveated subset 
of the video image data selected by a foveation function. 

14. The system of claim 13 in which the foveation function is a 
function of a current gaze point. 

15. The system of claim 13 in which the foveated 
transformational coding of the video image data includes threshold 
filtering of image data derived from the video image data. 

16. The system of claim 15 in which the threshold filtering of the 
derived image data corresponds to a human perception response to the 
derived image data. 

17. The system of claim 15 in which the threshold filtering is 
based upon a human perception spatial frequency response function to the 
image derived from the video image data and a human perception 
eccentricity visual response function to the image derived from the video 
image data. 

18. A method for compression of video image data comprising the 
steps of: 

acquiring digitized video image data; and 
performing by a computer system, real-time foveated 
transformational coding of the digitized video image data. 

19. The method of claim ,18 in which the real-time foveated 
transformational coding of the digitized video image data includes 
manipulation of a subset of the digitized video image data selected by a 
first foveation function. 
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20. The method of claim 19 in which the first foveation function 
corresponds to a first human perception response to first image data 
derived from the digitized video image data. 

21. The method of claim 19 in which the first foveation function 
corresponds to a first human perception spatial frequency response to the 
first image data and a first human perception eccentricity visual response 
to the first image data. 

22. The method of claim 20 or 21 in which the first foveation 
function is a function of a current gaze point. 

23. The method of claim 18 in which the real-time foveated 
transformational coding of the video image data includes threshold 
filtering of second image data derived from the video image data. 

24. The method of claim 23 in which the threshold filtering of 
the second image data is based upon a second foveation function. 

25. The method of claim 24 in which the second foveation 
function corresponds to a second human perception response to the second 
image data. 

26. The system of claim 24 in which the second foveation 
function corresponds to a second human perception spatial frequency 
response to the second image data and a second human perception 
eccentricity visual response function to the second image data. 

27. The system of claim 25 or 26 in which the second foveation 
function is based upon a current gaze point. 

28. A method for compression of video image data comprising the 
steps of: 
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acquiring digitized video image data; and 
performing by a computer system, foveated transformational 
coding of the digitized video image data; wherein the foveated 
transformational coding of the digitized video image data consists 
of foveated Laplacian pyramidal transformational coding, foveated 
wavelet pyramidal transformational coding, or foveated block 
discrete cosine transformational coding. 

29. The method of claim 28 in which the step of performing 
foveated transformational coding of the digitized video data is performed 
in real-time. 

30. The method of claim 28 in which the foveated Laplacian 
pyramidal transformational coding of the digitized video image data 
comprises the steps of: 

reducing the digitized video image data; 

expanding a foveated subset of the reduced digitized video 
image data; and 

subtracting the expanded foveated subset of the reduced 
digitized video image data to obtain difference video image data. 

31. The method of claim 30 in which the foveated subset of the 
reduced digitized video image data is determined by a first foveation 
function that correlates to a first human perception spatial frequency 
response function to the reduced video image data image and a first 
human perception eccentricity visual response function to the reduced 
video image data. 

32. The method of claim 31 in which the first foveation function 
is a function of a current gaze point. 

33. The method of claim 31 further comprising the steps of: 
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thresholding the difference video image data to obtain 
thresholded difference video image data; and 

quantizing the thresholded difference video image data to 
obtain quantized thresholded difference video image data. 

34. The method of claim 33 further comprising the step of . 
compressing the quantized thresholded difference video image data. 

35. The method of claim 33 in which the step of thresholding the 
difference video image data is determined by a second foveation function 
that correlates to a second human perception spatial frequency response 
function to the difference video image data image and a second human 
perception eccentricity visual response function to the difference video 
image data. 

36. The method of claim 35 in which the second foveation 
function is a function of the current gaze point. 

37. The method of claim 35 in which the first foveation function 
and the second foveation function are identical. 

38. The method of claim 36 in which the first foveation function 
and the second foveation function are identical. 

39. The method of claim 28 in which the foveated block discrete 
cosine transformational coding of the digitized video image data comprises 
the steps of: 

dividing the digitized video image data into non-overlapping 
blocks; 

applying a discrete cosine transform to each block to obtain 
transformed video image data; 

thresholding the transformed video image data with a 
foveation function to result in thresholded video image data; and 
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quantizing the thresholded video image data. 

40. The method of claim 39 in which the foveation function 
correlates to a human perception spatial frequency response function to 
the transformed video image data and a human perception eccentricity 
visual response function to the transformed video image data. 

41. The method of claim 39 in which the foveation function is a 
function of a current gaze point. 

42. The method of claim 39 further comprising the step of 
compressing the quantized thresholded difference video image data. 

43. The method of claim 28 in which the foveated wavelet 
pyramidal transformational coding of the digitized video image data 
comprises the steps of: 

convolving with a plurality of filtering kernals over the 
digitized video image to obtain plural subband images, the 
convolutions comprising high-pass convolution over a foveated 
subset of the digitized video image, the foveated subset selected by 
a first foveation function; and 

thresholding the plural subband images with a second 
foevation function to obtain plural thresholded subband images. 

44. The method of claim 43 further comprising the steps of: 
quantizing the plural thresholded subband images. 

45. A method of compressing a recorded video sequence, the 
method comprising the steps of: 

exposing an observer to the recorded video sequence; 

measuring periodically where the observer fixates when 
exposed to the recorded video sequence to obtain a likely gaze point 
for each period; 
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coding the recorded video sequence according to a foveation 
transformational coding employing the likely gaze point in each 
period. 

46. A method of compressing a recorded video sequence, the 
method comprising the steps of: 

exposing a group of observers to the recorded video sequence; 

measuring periodically where the observers fixate when 
exposed to the recorded video sequence to obtain a statistical likely 
gaze point for each period; and 

coding the recorded video sequence according to a 
transformational coding employing the statistical likely gaze point 
for each period. 

47. A method of displaying a compressed video sequence 
comprising the step of displaying a recorded video sequence compressed 
according to the method of claim 45. 

48. A method of displaying a compressed video sequence 
comprising the step of displaying a recorded video sequence compressed 
according to the method of claim 46. 

49. A method of compressing a recorded video image, the method 
comprising the steps of: 

exposing an observer to the recorded video image; 

measuring where the observer fixates when exposed to the 
recorded video image to obtain a likely gaze point; 

coding the recorded video image according to a foveation 
transformational coding employing the likely gaze point as a 
foveation function. 
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